Tuesday, August 22, 2017

IoT Time Series Data with InfluxDB and Knowi



InfluxDB is a Time Series NoSQL database specifically built for use cases centered around IoT/sensor data. This is a brief introduction into InfluxDB as well as analytics on InfluxDB using Knowi.  

IoT

Internet of Things -- the umbrella term for network aware devices, wearables, home, cars and other connected devices -- appears to be shedding its hype and starting to coming of age. The projections are certainly eye-popping, with an expected 50-200 billion connected devices by 2020, $1.7 trillion in spending, and 173 million wearable devices by 2019.

Given the volume and velocity of sensor data, NoSQL databases are often the preferred option to storing sensor data. Commonly used datastores include Cassandra, HDFS,  Graphite and more recently, Druid and InfluxDB.  

InfluxDB

InfluxDB is an open source database, created and supported by the InfluxData team. It can be downloaded here.

Highlights:

InfluxDB does not require a schema defined upfront. Queries are enabled using a nice SQL-like syntax with easy extensions for time-based queries. Write and Query API’s are exposed through a REST API.
Given its focus on time series databases, data is always stored with a timestamp component,  which acts like a primary index.

It uses a Line Protocol format, with measurements (like a SQL table), tags (like an indexed column) and fields (an unindexed column)  

Data Format:
<measurement>[,<tag-key>=<tag-value>...] <field-key>=<field-value>[,<field2-key>=<field2-value>...] [unix-nano-timestamp]

Example:
stock,symbol=AAPL bid=127.46,ask=127.48 1434067467100293230

In the example:
Stock is a measurement
Symbol is a tag
Bid and ask are fields.
The number field is the Unix timestamp in nanoseconds (optional: InfluxDB will auto insert time for the record)

Setup:

InfluxDB is straightforward to download and setup.

a. Download the appropriate package here.
b. Start the service.

To import some data:
Download test data: curl https://s3-us-west-1.amazonaws.com/noaa.water.database.0.9/NOAA_data.txt > NOAA_data.txt
(Water level data from the National Oceanic and Atmospheric Administration)

Insert: influx -import -path=NOAA_data.txt -precision=s

Queries:

Queries can be executed against InfluxDB using InfluxQL, a SQL- like syntax to query data stored in InfluxDB. Example:

SELECT * FROM mydb WHERE time > now() - 1d

This returns all tags & fields for the mydb measurement for last day.  

Analytics:

InfluxData provides Chronograf, a visualization tool to interface with InfluxDB. This might be an option to consider for time series charts if you are planning to store all your data in InfluxDB alone.   

Knowi provides native integration into InfluxDB to be able to query, aggregate, visualize, analyze data from InfluxDB, but also allows multi-datasource joins it with your polyglot persistence architectures (data across various NoSQL, SQL, REST API’s and files), all the way to produce interactive dashboards for non-technical users.

To see InfluxQL in action:

  1. Go to InfluxDB Instant Visualization page at Knowi. It’s configured with a live InfluxDB database and InfluxDB queries.
Note: The query generator section can be used to discover measurements, tags & fields and build basic InfluxQL dynamically.


  1. Click on Show me for instant visualizations.
  2. Experiment with your own time series InfluxQL queries against the data.

Enjoy!

Resources


Monday, August 21, 2017

How Lemonaid Health Monitors Performance with Knowi

Lemonaid Health Review of Knowi Our friends at Lemonaid Health use Knowi to manage their day-to-day operations and deliver business performance reporting across their teams and provider partners. Here is a little summary of their story.  

If you are not familiar with Lemonaid Health, they are an innovator in telehealth services. As of April 2017, their team of doctors treat things like sinus infections, UTIs, acne, acid reflux or write prescriptions for birth control pills.   Their mission is to make it easy to access affordable, high-quality healthcare.

THE CHALLENGE:   As new health services or new States roll out, understanding service performance in near real-time is critical for Lemonaid Health. They had two challenges: 1) ensuring people had the latest information and 2) heavy reliance on the engineering team to create and run SQL queries and then transcribe the results to Excel spreadsheets. 



“We fall under HIPAA so a cloud-based solution had to have the levels of security we need to be sure our patient data was protected,” says Simon Williams, Chief Technology Officer at Lemonaid Health.


Lemonaid began with using a combination of Excel and native SQL queries but found this did not provide the near real-time performance reporting they wanted and was too taxing on engineering resources who had other priorities. 

THE SOLUTION:  Lemonaid selected Knowi “because it was cloud-based and the fact the data would stay on premise.” says Williams.  Using hybrid deployment of Knowi enabled a cloud implementation of Knowi connected to on-premise instances of MySQL and MongoDB. Lemonaid’s commercial partners access Knowi directly, but filters limit their view to only their data.  Within the dashboards, their partners have full capabilities to run ad-hoc queries, change filters or visualization types and save them as custom reports. In addition to direct access for near real-time reporting, Lemonaid extensively uses Knowi’s sharing capabilities to send out PDF reports to offline users, ensuring everyone, company-wide, as access to up to the moment performance data.  This flexibility in data delivery methods was essential in transitioning the organization to be data-driven.

THE RESULTS:  The initial goal was of creating a company-wide unified view of all business performance data was quickly achieved.  “To be able to put up some real-time dashboards that showed what was going on at any point in time was key business information,” says Simon Williams.   “… [It] helped create a great buzz in the office as we displayed it on a large screen and could see counters going up as we served new patients." 

Once the engineering team built some foundational queries, “it was then easy for the business users to go in and look at the data however they wanted by State, service, doctor, day.”  The Lemonaid engineering team did not have to dedicate any additional time to generating reports or views.  Business users were taught to fish for the data themselves with Knowi.

Lemonaid quickly recognized the tremendous value of the data they collected for their partners, and they used Knowi to build and deliver a new class of data products for their commercial partners. Their partners now have near real-time data about how they are servicing their patients, demographics information about those patients and patient and service volume by State. 


To read or download Lemonaid's full story, head over to our resources page here.