Tuesday, August 22, 2017

IoT Time Series Data with InfluxDB and Knowi



InfluxDB is a Time Series NoSQL database specifically built for use cases centered around IoT/sensor data. This is a brief introduction into InfluxDB as well as analytics on InfluxDB using Knowi.  

IoT

Internet of Things -- the umbrella term for network aware devices, wearables, home, cars and other connected devices -- appears to be shedding its hype and starting to coming of age. The projections are certainly eye-popping, with an expected 50-200 billion connected devices by 2020, $1.7 trillion in spending, and 173 million wearable devices by 2019.

Given the volume and velocity of sensor data, NoSQL databases are often the preferred option to storing sensor data. Commonly used datastores include Cassandra, HDFS,  Graphite and more recently, Druid and InfluxDB.  

InfluxDB

InfluxDB is an open source database, created and supported by the InfluxData team. It can be downloaded here.

Highlights:

InfluxDB does not require a schema defined upfront. Queries are enabled using a nice SQL-like syntax with easy extensions for time-based queries. Write and Query API’s are exposed through a REST API.
Given its focus on time series databases, data is always stored with a timestamp component,  which acts like a primary index.

It uses a Line Protocol format, with measurements (like a SQL table), tags (like an indexed column) and fields (an unindexed column)  

Data Format:
<measurement>[,<tag-key>=<tag-value>...] <field-key>=<field-value>[,<field2-key>=<field2-value>...] [unix-nano-timestamp]

Example:
stock,symbol=AAPL bid=127.46,ask=127.48 1434067467100293230

In the example:
Stock is a measurement
Symbol is a tag
Bid and ask are fields.
The number field is the Unix timestamp in nanoseconds (optional: InfluxDB will auto insert time for the record)

Setup:

InfluxDB is straightforward to download and setup.

a. Download the appropriate package here.
b. Start the service.

To import some data:
Download test data: curl https://s3-us-west-1.amazonaws.com/noaa.water.database.0.9/NOAA_data.txt > NOAA_data.txt
(Water level data from the National Oceanic and Atmospheric Administration)

Insert: influx -import -path=NOAA_data.txt -precision=s

Queries:

Queries can be executed against InfluxDB using InfluxQL, a SQL- like syntax to query data stored in InfluxDB. Example:

SELECT * FROM mydb WHERE time > now() - 1d

This returns all tags & fields for the mydb measurement for last day.  

Analytics:

InfluxData provides Chronograf, a visualization tool to interface with InfluxDB. This might be an option to consider for time series charts if you are planning to store all your data in InfluxDB alone.   

Knowi provides native integration into InfluxDB to be able to query, aggregate, visualize, analyze data from InfluxDB, but also allows multi-datasource joins it with your polyglot persistence architectures (data across various NoSQL, SQL, REST API’s and files), all the way to produce interactive dashboards for non-technical users.

To see InfluxQL in action:

  1. Go to InfluxDB Instant Visualization page at Knowi. It’s configured with a live InfluxDB database and InfluxDB queries.
Note: The query generator section can be used to discover measurements, tags & fields and build basic InfluxQL dynamically.


  1. Click on Show me for instant visualizations.
  2. Experiment with your own time series InfluxQL queries against the data.

Enjoy!

Resources


Monday, August 21, 2017

How Lemonaid Health Optimizes Performance of Healthcare Services with Knowi

Lemonaid Health Review of Knowi Our friends at Lemonaid Health use Knowi to manage their day-to-day operations and deliver business performance reporting across their teams and provider partners. Here is a little summary of their story.  

If you are not familiar with Lemonaid Health, they are an innovator in telehealth services. As of April 2017, their team of doctors treat things like sinus infections, UTIs, acne, acid reflux or write prescriptions for birth control pills.   Their mission is to make it easy to access affordable, high-quality healthcare.

THE CHALLENGE:   As new health services or new States roll out, understanding service performance in near real-time is critical for Lemonaid Health. They had two challenges: 1) ensuring people had the latest information and 2) heavy reliance on the engineering team to create and run SQL queries and then transcribe the results to Excel spreadsheets. 



“We fall under HIPAA so a cloud-based solution had to have the levels of security we need to be sure our patient data was protected,” says Simon Williams, Chief Technology Officer at Lemonaid Health.


Lemonaid began with using a combination of Excel and native SQL queries but found this did not provide the near real-time performance reporting they wanted and was too taxing on engineering resources who had other priorities. 

THE SOLUTION:  Lemonaid selected Knowi “because it was cloud-based and the fact the data would stay on premise.” says Williams.  Using hybrid deployment of Knowi enabled a cloud implementation of Knowi connected to on-premise instances of MySQL and MongoDB. Lemonaid’s commercial partners access Knowi directly, but filters limit their view to only their data.  Within the dashboards, their partners have full capabilities to run ad-hoc queries, change filters or visualization types and save them as custom reports. In addition to direct access for near real-time reporting, Lemonaid extensively uses Knowi’s sharing capabilities to send out PDF reports to offline users, ensuring everyone, company-wide, as access to up to the moment performance data.  This flexibility in data delivery methods was essential in transitioning the organization to be data-driven.

THE RESULTS:  The initial goal was of creating a company-wide unified view of all business performance data was quickly achieved.  “To be able to put up some real-time dashboards that showed what was going on at any point in time was key business information,” says Simon Williams.   “… [It] helped create a great buzz in the office as we displayed it on a large screen and could see counters going up as we served new patients." 

Once the engineering team built some foundational queries, “it was then easy for the business users to go in and look at the data however they wanted by State, service, doctor, day.”  The Lemonaid engineering team did not have to dedicate any additional time to generating reports or views.  Business users were taught to fish for the data themselves with Knowi.

Lemonaid quickly recognized the tremendous value of the data they collected for their partners, and they used Knowi to build and deliver a new class of data products for their commercial partners. Their partners now have near real-time data about how they are servicing their patients, demographics information about those patients and patient and service volume by State. 


To read or download Lemonaid's full story, head over to our resources page here.  

Monday, July 17, 2017

Add Predictive Analytics on Your Dataset in 3 Steps

Knowi Predictive Analytics brings machine learning capabilities to every Data Engineer.

Add Predictive Analytics on Your Datasets
Integrate machine learning into your analytics workflows and drive data directed actions.

The platform provides two options:
  • Out-of-the-box predictive analytics capabilities that test a dataset against a variety of forecasting models to determine the model best suited to the data,  with the least Sum of Absolute Errors (SAE).  
  • Built-in Predictive and Machine Learning algorithms that can be plugged into data workflows.
This post focuses on the first option above that takes a hands-on look at how it works.  We'll take monthly stock prices for Amazon to determine predicted values over a three month period starting in July in a few simple steps. No signup is required to follow along.  

Models used include:
  • Simple Exponential, Double Exponential, Triple Exponential Smoothing Models
  • Moving Averages and Weighted Moving Averages
  • Naive Forecasting Model
  • Regression and Polynomial Regression Model
  • Multiple Linear Regression Model

The monthly stock prices looks like this:

Date, Price
06/01/16, 719.14
05/01/16, 683.85
04/01/16, 598.00
03/01/16, 579.00
02/01/16, 574.81
01/05/16, 633.79
12/01/15, 679.36
11/2/15, 628.35
10/1/15, 625.90
9/1/15, 511.89
8/3/15, 512.89
7/1/15, 536.15
6/1/15, 434.09
5/1/15, 429.23
4/1/15, 421.78
3/2/15, 372.10
2/2/15, 380.16
1/2/15, 354.53

Steps:

1. Copy and paste the above dataset into https://www.knowi.com/csv-json-files-analytics.



2. Click on Show me. The data will be parsed and visualized immediately. 

3. To perform predictions:
   i. Click on Analyze from the menu of the time series chart. This opens up an Analysis mode.
  ii. Drag date into the Grouping field. 
  iii. Click on 'Add a derived Field' option. Enter a name ("Predictions", for example) and in the operation, enter predict(price,date,07/01/2016,1m,3). This will choose the best model based on historical accuracy of the model to determine the projected prices over a three month period, on a monthly basis. 




That's it! In a few simple steps, you can apply predictive analytics on any of your own datasets. Enjoy!

Get started with Knowi Predictive Analytics at https://knowi.com.  Try it out free.

Additional Machine Learning Resources:
Predictive Analysis docs: https://docs.knowi.com/hc/en-us/articles/115006493407-Predictive-Analytics-Forecasting
Advanced Machine Learning (AI) Capabilities: https://knowi.com/adaptive-intelligence-machine-learning
All Documentation: https://docs.knowi.com





Monday, July 10, 2017

How Boku Successfully Transitioned to a Data-Driven Enterprise Using Knowi


Boku Testimonial - KnowiWe had a chance to sit down with our friends at Boku recently and talk about how they are using Knowi to provide business analytics across all their teams, including product, accounts, and business development, and have recently extended their analytics as value-added product offerings to their partners and customers.

If you are not familiar with Boku Inc. they are a global mobile payments platform enabling customers to pay for goods and services using their mobile phone number. 

THEIR CHALLENGE is to serve their growing merchant client base, the account management and business development teams needed access to a complete view of their client accounts status but the only way to access that data was to query the database.  


As a payment platform serving dozens of countries, Boku analyzes terabytes of data and the business needs results returned within seconds to serve their customers.  


THE SOLUTION in Boku’s case is a hybrid deployment of Knowi enabled a cloud implementation of Knowi connected to an on-premise instance of MySQL.  Query results are cached in Knowi’s Elastic Store, so visualizations are rendered within seconds providing an excellent user experience while ensuring the database performance is not impacted by query requests.


Boku also took advantage of Knowi’s ability to share and embed dashboards into data applications. Embedding dashboards eliminated the need for business users to have any expertise in SQL, a key goal for the team.  Business users simply go to their customized dashboard and use a drag & drop interface to run ad-hoc analysis on any dataset they are authorized to access.


RESULTS


 ------------
"Knowi’s ability to handle our large volumes of data and let everyone here at Boku to quickly access and share information was critical to becoming a data-driven organization.  By using Knowi to leverage our data company-wide, we were able to accelerate a number of strategic goals with our partners."

Mike Cahill, CTO, Boku
------------ 

Knowi was rapidly adopted company-wide to include the product management, finance, and operations teams.  Product managers use it to understand how customers are using the platform, tune the product and find ways for partners to save money.  The finance and operations teams use it to monitor company-wide key performance metrics. 

Because Boku was able to implement Knowi in place of an enterprise data warehouse strategy, they estimate up to a 70% reduction in their cost of information.  Additionally, the account and business development teams have increased upsell/cross-sell opportunities by at least 15% directly impacting the bottom-line.


Finally, the business intelligence team uncovered an opportunity to monetize Boku’s transactional data and create new data-driven product offerings to sell to partners who need the same level of transactional analytics that Boku is producing internally.  As a result, “our B.I. team that is normally a cost center can now generate revenue, so that’s a great bonus for the team” says Walter.


To read or download Boku's full story, head over to our resources page here.  


Thanks to the team at Boku for allowing us to tell their story!

Thursday, June 1, 2017

Knowi Product Update - Machine Learning; Enterprise Management API; New Name!

Since our Q1 Update, a few things have changed, namely our name!  You've probably already noticed that back in late April, we became Knowi.  We've been planning a name change for a while and with the latest set of product releases, the timing seemed right to make the change from Cloud9 Charts to Knowi.  We hope you like it.

Name changes aside, we are excited to tell you about a number of new features we added this past quarter.  We've been hard at work adding:

AI + BI for Modern Data

We're pretty proud to be the first in the market to combine all the power of our Business Intelligence platform for unstructured and structured data with integrated Artificial Intelligence. With AI + BI, you can blend hindsight and foresight to drive actions from your data for any number of use cases. Not only that, it also enables you to offer value added machine learning capabilities to your customers, for embedded use cases. We've integrated powerful machine learning capabilities into the platform including:
  • Built-in algorithms (you can also upload your own proprietary algorithms)
  • Evaluation statistics, like accuracy and precision, to help you determine best algorithm
  • Model training using historic data or upload a training dataset
  • Save the trained model and add it to any relevant query with a couple of clicks
  • Configure actions to be taken based on query results, including triggering a downstream application or sending a notification

This powerful new feature is currently in beta and we will be adding other algorithms to it over the next few weeks. Sign on or Sign up to try it out. To get more details, read our machine learning documentation.

Management API

We've developed a public Management API to help our enterprise customers manage various aspects of the Knowi platform. The API is RESTful allowing external services and apps to manage users and groups, datasources, queries, dashboards and widgets programmatically.

To get the details, read our API documentation which also includes code examples.

Fine-Grained User-Level Filters and Enterprise Access Controls

The combination of content filters, both at a user level and shared URL level, and query parameters allow admin users to limit user access to a subset of available data on the same dashboard or widget. Filters can be applied in one of two ways:
  • Filters are passed through as query parameters, at runtime, for direct queries
  • Filters are applied on top of the data returned by a query
Read more details about User Filters in our documentation.


Knowi admins now have more granular control to automate the permissions for sharing data assets (dashboards, queries, datasources, agents). By default, assets are private to the user, unless shared to other users or groups. Now, admins can configure assets so they are automatically shared within a group. Furthermore, each asset can be configured for granular read or edit access at a group or an individual user level.

For details on new access controls, please reference our permissions documentation.

Searchable Knowledge Base and Introducing Lorraine Williams, Head of Success

We are excited to introduce Lorraine to you. Lorraine heads up product management and customer success. We've known Lorraine for a while as a customer, and she comes to us with many years of product and engineering management experience. She's also really good at math having just completed her MS in Analytics.

Going forward, she will ensure our customers get the most out of the Knowi platform from day one. She also will work closely with the Engineering team to ensure we continue to deliver functionality important to our customers and our long-term product vision.

Our customers will see some immediate changes in the way we communicate product updates, including the posting of weekly release notices in our new searchable Knowi Knowledge Base. Yay, Lorraine!

OTHER COOL STUFFF

NEW VISUALIZATIONS
We've added Waterfall Charts, Box-Plot Diagrams and the ability to embed an external Webpage as a widget in your dashboard. You can also now customize the CSS of dashboards for custom branding
CATEGORIZING WIDGETS AND DASHBOARDS
To help manage large numbers of widgets and dashboards, we've added the ability to categorize or "tag" widgets and dashboards so you can group related items together. When saving a new widget or dashboard, you can create a new category (names will be auto-suggested) or select an existing category from a list.
DATA EXPLORER
To make it even easier for you to understand, discover, and explore your multi-structured and unstructured data sources, Data Explorer performs a "schema on read" to sample your data and glean from it relationships and any possible structure. You can drilldown into nested data and drag and drop fields into query parameters.

Data Explorer is available from the Query Builder page and automatically appears on the left side as soon as you select a datasource.
ENHANCED PDF REPORTS
For PDF reporting, we've enhanced the layout so if a large table/grid is part of the dashboard the full table will be render in the PDF. We've also added headers and footers to make multi-page reports easier to read.
NEW DATASOURCE
We've added support for PRESTO. Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.
Try Knowi Free 
https://www.knowi.com/

Tuesday, May 9, 2017

We're Hiring!

We are actively looking for Sales Development Representatives and have interviewed a number of candidates to-date but are still looking.  I'd actually like to share the experience we had with one of them to maybe help those of you looking to stand out from the pack.

First, this candidate found us.  She researched company profiles looking for the type of company she wanted to join.  She then found our CEO on LinkedIn and emailed him describing how she be an asset to us and asked to have a quick phone conversation.

She is a graduate of alwayshired.com which is a Sales Bootcamp program, at least according to their website.  We have no affiliation with them, but if this candidate is a good representation of the quality of their graduates, then they seem to know what they are doing.

Anyhow, we did the regular interview process eventually offered her a position.  We knew she was pretty green, so why did we offer her a job?
  • She hustled to get an interview and, in doing so, demonstrated her sales methodology
  • She confidently pitched herself but without being obviously boastful or arrogant
  • She was genuine and shared personal experiences and stories (not in a weird way)
  • She had a 100% positive attitude
  • She demonstrated a high level of personal integrity and character
Cloud9 Charts is Knowi
Like a good SDR, she had secured a few leads in her pipeline, and we were just one of them. Unfortunately, for us, we lost out to another company closer to her home.

The following week, we changed our name from Cloud9 Charts to Knowi.  We were totally surprised to see her walk through the door carrying a cake.  Yes, she bought us a cake!  Now, buying us a cake to celebrate our name change after not taking our job offer is arguably going above and beyond but who doesn't love cake?

If you haven't picked up on it yet, it's your character and attitude that makes you stand out in an interview.  You should assume everyone has similar skills and experience, so it's about allowing people to see who you are in a genuine way.  It sounds simple but is not.  That is why, as a hiring manager, when you see it that person immediately stands out.

Thanks, Anna Aleksenko.  We loved the cake!

We've are still looking for the next SDR superstar in the making.  Are you the one?  You can see the full job description and submit your application here:  https://www.localwise.com/job/17025-sales-development-representative/15365-knowi-oakland-ca

Our Open Positions:

Enterprise Account Executives and Sales Development Specialists
We are looking for unproven legends in the making. We are a startup poised for exponential growth this year, and you'll play a crucial role in making that happen.

You'll work directly with our founding team with deep experience in the space. Knowledge and skills you learn here will put you in good stead for years to come.

If you can talk the talk and walk the walk, you'll have unprecedented growth opportunities as the company scales up.

What you need:
  • Lots of ambition, hunger, and a deeply rooted will to succeed.
  • "Whatever it takes" attitude to not only bring in new customers but most importantly, make them successful.
  • Minimum 1-2+ years of sales experience, preferably in a SaaS role for the Sales Development Specialist. 5+ years of experience for Enterprise Account Executives.
  • An upstanding individual with unquestionable honesty, integrity and leadership potential.
  • Quick learner with the intellect and technical aptitude to distil complex solutions with the ability to communicate that succinctly to technologists and executives.
  • We prize an entrepreneurial spirit with the right make-up to inspire and lead as we grow.
Think you have what it takes? Become that legend. We'd love to hear from you: sales@knowi.com.

Engineering & Design
We are currently hiring Java Engineers, UI and UX Designers and would love to hear from you. Some samples of your code/designs or any of your projects in GitHub would also be helpful. Email us at info@knowi.com.

Support & Customer Engineering
  • Prior Experience in the data space.
  • The right Aptitude and Attitude.
  • Work closely with the founding team and sales.
  • Quickly come up to speed on our product, with the ability to communicate and distil complex technical questions to suggest solutions.
  • Minimum of 4 years of experience in a customer facing technical role.
  • Solutions Architects
  • We are looking for a unique mix of technical prowess, customer facing skills and a magnetic personality to lead pre-sales and post-sales engineering. 
  • Prior Experience in the data space.
  • Proficient in SQL, NoSQL, and various database technologies.
  • Prior experience in Analytics and BI a plus.
  • Work closely with the sales team, product, and engineering.
  • Provide delightful customer experiences and solve their technical problems.
  • At least 5 years of experience in a customer facing technical role.
If you are interested in this role, email us at info@knowi.com.

Thursday, April 27, 2017

Knowi on MongoDB Atlas - Tutorial

This is a 10 minute, hands-on tutorial of setting up connectivity to MongoDB Atlas and building visualizations from it using Knowi. The end result is a dashboard of restaurants near the NYC area. In a previous post, we reviewed MongoDB Atlas.



CONNECTING

This assumes that you have an Atlas account setup. If you don't, go to the MongoDB Atlas page and sign up for a free sandbox account. (More details on setting up your database on Atlas here).  

  1. Importing data into Atlas: 
b) Use MongoHub or mongoimport to import the JSON file into a collection.  
   
The JSON structure looks like this, with some nested elements:

{
  "borough": "Bronx",
  "cuisine": "Bakery",
  "name": "Morris Park Bake Shop",
  "restaurant_id": "30075445",
  "address": {
    "building": "1007",
    "coord": [
      -73.856077,
      40.848447
    ],
    "street": "Morris Park Ave",
    "zipcode": "10462"
  },
  "grades": [
    {
      "date": {
        "$date": 1393804800000
      },
      "grade": "A",
      "score": 2
    }
  ]
}
2. Sign up for a free Knowi account.
3. Create a MongoDB datasource connection:
  • Whitelist our IP addresses in Atlas to enable connectivity. (Alternatively, An on-premise way to set up connectivity is also available. See agent docs for more details.)
  • Create a new connection in Knowi via the Datasources icon --> New Datasource --> MongoDB.
  • Add the replicaset hosts into into the Host(s) section. Example: c9demo-shard-00-00-zito5.mongodb.net:27017,c9demo-shard-00-01-zito5.mongodb.net:27017,c9demo-shard-00-02-zito5.mongodb.net:27017
  • Leave the port empty, if you already have ports in the hosts section; specify database to connect to, along with the user and password.
  • Under the 'Database Properties' field, add the following: ssl=true&authSource=admin&replicaSet=<yourRelplicaSetName>
  • Use the 'Test Connection' button to ensure that the connection is successful. Save.
Now the fun begins. Let's bring this data to life.



GENERATING QUERIES



After the datasource is saved, click on the Configure Queries link. This will open up the query page. Let's start with a count of restaurants by borough by cuisine.

  • Open up the Query Generator.
  • Collections should be automatically populated with the collections from Atlas. Choose Restaurant collection. This will trigger our field discovery process to determine fields in the collection.
  • Add borough and cuisine to the dimensions dropdown.
  • Add _id into the Metrics section; click on it to add an count aggregation to it.
  • Notice the auto generated MongoDB query with aggregations.
  • Click on Preview to instantly preview the results.

VISUALIZATIONS

Create a dashboard and add our newly created report/widget into it:


Set up a stacked bar chart with Borough in the X axis, cuisine in the Y axis.




Set up a few filters (via the filter icon):


In a few simple steps, you have your first visualization from data in Atlas.

DRILLDOWNS

Now, let's take this a step further with a drilldown to produce a cluster map of restaurants  for a given 'borough' & 'cuisine.'





Add another query, this time without aggregations and including the nested address object.


Drill into the address object to drag & drop the 'coord' array into the ad hoc analysis grid, along with the 'borough', 'cuisine' and 'name':

 


Set it up as a Geo Marker Cluster Visualization. Note that the coord nested array is automatically unwound as lat/long coordinates on the map.




On Visualization settings, set the Name to the restaurant name, and center the map to NYC (Lat/Long of 40, -74). Save the new query/Widget.


Setup a drilldown from our parent bar chart into the map.






Click on any point on the bar chart to get a cluster map of restaurants for the clicked combination 'borough' & 'cuisine':



SUMMARY
To summarize, we imported data into an Atlas cluster, connected to it using Knowi, generated a few native MongoDB queries on it and set up a few visualizations from it.

Now it's your turn to bring your own data to life!

RESOURCES