Posted on

Modern Data Acquisition Software at DAQifi

Developing a Great User Experience via Desktop App

Data acquisition (DAQ) systems on the market are cumbersome, hard to use, require a wired connection to a computer, and have antiquated software. Even worse, most DAQ software only allow the user to connect to a single DAQ device. Simply put, current DAQ systems provide a poor user experience. DAQifi is solving this problem by making user experience a priority. One opportunity to focus on user experience is through the desktop application.

In desktop applications, a modern user interface is important. Gone are the old, ugly interfaces of Windows 98, which most DAQ software still seem to levitate towards. DAQifi has developed a modern desktop application with the user in mind. The user easily and intuitively adds multiple DAQifi devices to a single computer, configures channels, and starts logging data in under a minute. DAQifi devices are connected wirelessly to the user’s computer through the user’s already existing wireless network. This means the user can have flexibility on where to place the DAQifi devices; no need to run long wires to the computer.

DAQifi Desktop App

Another key to great user experience is listening to the user’s needs. DAQifi developed a minimum viable product (MVP) which included basic features users would expect such as: real-time data logging, archiving the data, and exporting the data to Excel so it can be analyzed. The MVP is the foundation for which features are added to through understanding customer’s data acquisition needs. In beta testing, DAQifi quickly validated that being able to downsample and average data was a key for customers to manage large amounts of data. This feature was added to the MVP and is now widely used.

Recently, DAQifi has learned that scaling inputs is extremely valuable. As such, input scaling is being included in the next release of the desktop application (v0.8) which will be available within the next couple of weeks. The current version of the software (v0.7) can be found at https://www.daqifi.com/support.

Posted on

Simple User Reset Circuit for Microcontroller

The RST pin of your micro controller is not the most exciting of the bunch, but is one of the most important (and potentially dastardly).  When I dive into a design I’m concerned with the big picture and what grandiose tasks my mighty micro is destined to accomplish… My enthusiastic efforts when beginning a project often cause me to overlook some details until a couple weeks or revisions later.

Only then do I begin to examine the small details of the user interface.  For example, when my iPhone freezes (yes, I own one Apple product), I can expect to hold down the power button for a few seconds and it will confirm I want to shutdown my phone.  This operation has become a loose convention in the consumer world.

Likewise, I want my micro to behave in a similar fashion. I found there are many dedicated chunks of silicone for this exact task and to my surprise, they command a decent price for such a simple process ($0.76 FT7521L6X or $1.36 TPL5010DDCT). I tried a few out of curiosity and they worked as expected. One prevailing feature of such components were their incredibly small footprint. Simple operation generally means small real estate. However many of the products I encountered were obviously designed for mobile devices: phones especially.  Fitting this functionality in a 1mm X 1mm space is not my concern. I want reliable operation for a budget price…

My next revision fell back to basic circuits knowledge: the RC circuit. I want to trigger a reset after a time delay and a resistor and capacitor could easily do that. So below, I implement a 10uF cap with a 1.8M resistor (with an extra couple diodes D1 & D2 and mosfet M1 to provide other functionality specific to my application). As the switch closes, the cap slowly charges through the resistor. Once the voltage reaches the threshold of the reset transistor, my micro resets. Circuits 101 at work – certainly not rocket surgery!

Simple RC Microcontroller User Reset Circuit (LT Spice Simulation)
Simple RC Microcontroller User Reset Circuit (LT Spice Simulation)

This is great. My slightly overlooked feature is addressed and I can move on. But I always like to test all use cases… What if I press the button many times in a row? Well, obviously, the cap will charge incrementally upon each press. After enough cumulative charge built up, my simple duo would reset my micro. However this is not the intended operation. Perhaps the user will also use my button to set other features/modes of my project? I don’t want it to reset inadvertently after a few presses!

Simple RC Microcontroller User Reset Circuit with Unintended Operation (LT Spice Simulation)
Simple RC Microcontroller User Reset Circuit with Unintended Operation (LT Spice Simulation)

One tweak provides a simple solution to my problem: charge the cap slowly when the button is pressed, but discharge quickly when not pressed. To accomplish this my circuit simply needs a diode across the resistor.

Simple RC Microcontroller User Reset Circuit with Discharge Diode (LT Spice Simulation)
Simple RC Microcontroller User Reset Circuit with Discharge Diode (LT Spice Simulation)

Now my reset circuit behaves just like the iPhone, without the cost or hassle of a 1mm x 1mm package and at ~1/6 the cost!

Posted on

Bombarded Database = Angry Database

A big challenge in designing DAQifi’s desktop app is the fact that we have to save a large amount of data and we have to do it quickly. Our Nyquist board has up to 20 input channels.  Each channel can be sampled at 1000Hz.  That’s 20,000 individual data samples per second streaming wirelessly from the board…over a million individual samples per minute!

FAILED ATTEMPT #1

At first we took the naive approach of saving data to the database as soon as we received it.  Our logging code looked similar to the code below.

using (LoggingContext context = new LoggingContext())
{
     //Add data sample
     context.Samples.Add(channel.ActiveSample);
     
     //Commit changes to the database
     context.SaveChanges();
}

This worked well for low sampling rates, but completely fell apart at higher rates.  The problem is that the database didn’t appreciate being bombarded 20,000 times a second and I don’t blame it!  It simply couldn’t keep up with that many transactions that quickly.

FAILED ATTEMPT #2

After much research on the struggles and tribulations others have had, a common theme emerged.  It’s best to do fewer but larger transactions.  Sure enough, the internet was right!  Instead of immediately saving to the database, we buffered data in memory and wrote to the database once a second.  This drastically improved our throughput.  We even found some tips and tricks about turning off AutoDetectChanges in Entity Framework which helped speed things up.

if (_stopwatch.ElapsedMilliseconds > _lastUpdateMilliSeconds + 1000)
{
     using (LoggingContext context = new LoggingContext())
     {
          context.Configuration.AutoDetectChangesEnabled = false;
          
          //Add buffered data to the database
          context.Samples.AddRange(_buffer);
          
          //Clear the buffer
          _buffer.Clear();

          //Commit changes to the database
          context.SaveChanges();
          context.Configuration.AutoDetectChangesEnabled = true;
     }
}

Unfortunately, this still wasn’t good enough and had some pitfalls.  To begin with, data could be added to our buffer between writing to the database and clearing the buffer.  This means we could be loosing data…definitely undesirable in a data acquisition system!  And although our performance had increased, it still couldn’t keep up with the Nyquist at high speeds.

THIRD TIME’S A CHARM!

More research ensued and we discovered a Bulk Insert library for Entity Framework.  Bulk insert drastically improved the commit time to the database.  We also re-architected our consumer thread a bit to have its own temporary buffer.  Now data samples are from the main buffer and copy them to the temporary buffer.  This way, if data gets added between us committing data and clearing out the buffer, we’ll catch it on the next time around.  Our final solution looks similar to this:

private void Consumer()
{
     var samples = new List(); //temporary buffer
     var count = _buffer.Count; //get a snapshot of main buffer
     using (LoggingContext context = new LoggingContext())
     {
          //Remove samples from the main buffer
          for (int i = 0; i < count; i++)
          {
               DataSample sample;
               if (_buffer.TryTake(out sample)) samples.Add(sample);
          }

          //Save temporary buffer to database
          context.BulkInsert(samples);
          
          //Clear out the temporary buffer
          samples.Clear();
     }
}

IN SUMMARY

Overall we learned the following in dealing with database performance

  1. Databases don't like to be harassed. There is a limit to the number of transactions it can handle per second.  It's best to do fewer but larger transactions.  There is a balance though!  If you do really large transactions, you'll see a performance hit.
  2. Don't reinvent the wheel. There are some awesome libraries out there to help with inserting a lot of data in the the database at once.
  3. Be careful with data integrity. We implemented a double buffer system to ensure we don't lose any data from the data acquisition board.