blink detection

[mehaghiles]

i want to detect blinking eyes using your librarie but i don't know if it's possible or how to do this

[orgicus]

@mehaghiles I'm afraid you won't be able to do that with an Arduino/serial device (as the communication protocol doesn't support it)

You should be able to use their SDK on a computer.
(Nowadays there are small form factor PC computers like UpBoard if that's a restriction)

See issue #1 for details

[mehaghiles]

@orgicus Thank you for your response. 

It took a while for me but i finally found how to detect the blinking eye. I'm working on a project since months and i need to use both of blink detection and the attention value, unfortunatly i have some isues using your library with the code that i have. When I try to use the command "mindwave.update" nothing happens.

I was wondering if you could help me solving this problem.

[orgicus]

@mehaghiles Nice work on getting eye blink detection working.

I would need to understand which Serial stream you're using and how you have connected the wires between the Arduino and the Bluetooth module.

Can you post a minimal version of your sketch and a wiring diagram ?
(You can use the Insert Code (<>) button to format it)

[mehaghiles]

Here is the code i use to get the eye blinking detection, anyway it's not mine i found it after many researches on google. The pnly connexion i have is the bluetooth wich recieve data from the headset and is connected to Rx pin, it's almost the same connexion as the one on your librarie because i use both codes without changing the connexions. Anyway here is the code.
 

#define BAUDRATE 57600

boolean DEBUGOUTPUT  = false,piekDetected = false;

int Data[512] = {0};

int i = 0, n = 0 ;

int PiekP, PiekM;

long piekTime = 0;

int payloadLength = 0;

byte payloadData[64] = {0};

byte poorQuality = 200;
 

void setup()
{
  Serial.begin(57600);
   

}
 

byte ReadOneByte()
{

  int ByteRead;

  while (!Serial.available());

  ByteRead = Serial.read();
  return ByteRead;

}
void loop()

{

  long Hilf = 0;
  if (i >= 512)

    i = 0;
  if (ReadOneByte() == 170 && ReadOneByte() == 170)

  {

    payloadLength = ReadOneByte();

    if (payloadLength == 4)

    {

      if (ReadOneByte() == 128 && ReadOneByte() == 2)

       {

        Hilf =  ((long)ReadOneByte() * 256 + (long)ReadOneByte());

        if (Hilf > 32767)

          Hilf -= (long)(65535);

        Data[i] = (int)Hilf;
        ReadOneByte();
        PiekP += Data[(512 + i - 71) % 512];

        PiekP -= Data[(512 + i - 50 - 71) % 512];
        if ((PiekP > 3000) && (Data[(512 + i  - 70) % 512] < 0) && (PiekP < 13000))

        {

          if (millis() - piekTime > 100)

          { PiekM = 0;

         

            for (int j = 1; j <= 70; j++)

              PiekM +=  (int)(Data[(512 + i  + j - 70) % 512]);
 

            if (PiekM < -3000 && PiekM > -11000) {
              if ((millis() - piekTime) < 600)n++; else n = 1;
              if(poorQuality == 0)digitalWrite(6,HIGH);else digitalWrite(6,LOW);

              piekTime = millis();

              piekDetected = true;

            }

          }

        }

        i++;

       

        if (DEBUGOUTPUT && i == 512)

        {

          for (int j = 0; j < 512; j++)

          {Data[j];

            Serial.print(Data[j]);

            Serial.print(";");

          }

          Serial.println("");

        }

      }

}

    else if(payloadLength < 170){

      for (int k = 1; k < payloadLength; k++)

      if(ReadOneByte()== 2)poorQuality = ReadOneByte();

    }

  }

  if((millis()-piekTime)>700 && piekDetected == true)Serial.println(n),piekDetected = false;
}

[mehaghiles]

@orgicus i wish you'll find a sollution to add the eye blinking detection in your librairie or just find a way to use i in this program because i've tried so many times but it don't work att all

[orgicus]

Hi @mehaghiles

What you've sent is is confusing.
Does the code you posted above actually acurately and confidently detect eye blinking or does it appear to work every once in a while ?
I'm asking because according to the official NeuroSky documentation the Stream API (e.g. using Arduino + BT module) does not support eye blinking detection as I've mentioned before: http://developer.neurosky.com/docs/doku.php?id=thinkgear_communications_protocol#eye_blink_strength

My short answer is that you shouldn't attempt to get blink data with an Arduino and use a computer with bluetooth where you would run the official Mindwave SDK (ThinkGear API) to get all the data you need. It really depends on what your overall project is trying to acomplish and how you get there.

Given that I have no Mindwave headset to test, limited time and you are asking for support on functionality that is nor supported by this library this is the best I can do support additionally:

Where did you find this snippet ?
(It looks similar to original Arduino sample code(http://developer.neurosky.com/docs/doku.php?id=mindwave_mobile_and_arduino) however it is much messier)

The pnly connexion i have is the bluetooth wich recieve data from the headset and is connected to Rx pin,

By Rx pin do you mean Arduino pin 0 RX, a different SoftSerial pin ?
If it's difficult making a diagram perhaps posting a picture of how your Bluetooth module connects to your arduino would be easier ?
(My assumption at the moment is that you've got the wiring perfectly. It's also helpful to know if you can use hardware Serial at all or not (e.g. if you use Arduino RX/TX pins for the bluetooth module you can't use it with a Serial connection via USB to a computer at the same time))

At the moment I don't have a headset to test, however if you are 100% confident the code you have detects blinks and still want to use my library this what I'd advise:

1. Cleanup the code code above (format it, name variables correctly,
2. Try to debug/annotate/understand the code from PiekP += Data[(512 + i - 71) % 512]; to the block ending in piekDetected = true;
3. The library exposes the equivalent of Data[] in your code as .eeg(): See the EEG example for more info

Here is a rough attempt at merging your code above with the library please note that I can't test the code (without access to the hardware) and I have no guarantee that your code above works perfectly. This is more a guide on how you might want to approach this, not a working solution:

#include <Mindwave.h>

Mindwave mindwave;
int*     data;

int peakP, peakM;
int i = 0, n = 0;
long peakTime = 0;

boolean peakDetected = false;


void setup() {
  Serial.begin(MINDWAVE_BAUDRATE);
}

void updateBlink(){
  data = mindwave.eeg();

  for(int i = 0 ; i < MINDWAVE_EEG_SIZE; i++){
    // print debug data
    Serial.print(data[i]);
    if(i < MINDWAVE_EEG_SIZE) Serial.print('\t');
    // TODO: cleanup / debug / test
    peakP += data[(512 + i - 71) % 512];
    peakP -= data[(512 + i - 50 - 71) % 512];

    if( (peakP > 3000) && (data[(512 + i - 70) % 512] < 0) && (peakP < 130000)){

      if (millis() - peakTime > 100){
        peakM = 0;

        for (int j = 1; j <= 70; j++){
          peakM += (int)(data[(512 + i + j - 70) % 512]);
        }

        if (peakM < -3000 && peakM > -11000) {

          if ((millis() - peakTime) < 600)
            n++; 
          else 
            n = 1;
          
          if(mindwave.quality() == 1000)
            digitalWrite(6,HIGH);
          else 
            digitalWrite(6,LOW);

          peakTime = millis();

          peakDetected = true;

        }
      }
    }

  }

  Serial.println();

  if((millis() - peakTime) > 700 && peakDetected == true){
    Serial.println(n);
    peakDetected = false;
  }
  
}

void loop() {
  mindwave.update(Serial,updateBlink);
}

Rather trying to slap random bits of code without understanding them fully try to slow down a bit, break the problem down into smaller simpler chunks. Imagine what the code does line by line then user println() to test your assumptions then correct them or the code as needed.
(I used to try smashing programs together too when I was getting started with coding and once a program would pass a super basic level of complexity that wouldn't work anymore: it paid off understanding each program and writing a fresh clean solution using the experience).

Also check out this how to program guide.

Hope this helps,
George

[mehaghiles]

Thank tou for those advices i'll try to do what you said.
The code i have sent actually detects blinking perfectly i've tested it many times that's why i want to use it.
By rx pin i mean that Tx pin of the bluetooth (HC-05) is connected to the Rx pin of the arduino uno( the pin 0) you can see this on the joigned picture.
I've tried to fully understand the code i have sent but i'm not that good at coding yet so it's a bit hatd for me but with your advices and the code you sent to me i'll try to solve my probleme.

[mehaghiles]

@orgicus i have tried the code you wrote and it's not working. The original code print the times i'm blinking my eyes only when i blink my eyes. But yours update the values wether if there is no blinking detected. And with your code the printed datas ar the raw data received by the headset.
Here are two pictures to show you the results the first one is by the original code and the second one is by using yours.
I want to get the attention when a number of blinking is detected, for example 2 eyeblinking is detected i want to get the attention value ( with the commande mindwave.update and make it update ) to do an action.

[orgicus]

@mehaghiles Thanks for posting the circuit image: very helpul !
Great to see you can use Serial Monitor to debug.

As previously mentioned, my library supports the Mindwave standard ThinkGear API which doesn't include Blink detection. That being said: if it can be accurately extrapolated from the EEG data it would be nice to contribute that for other users as well.
The main issue is I am unable to commit the required time to properly debug/fix this, not to mention I don't have access to a headset.

At best, I may have a bit of time mid August and would need your time perhaps on remote calls to test/debug with an actual device.

If this timeline and level of commitment isn't feasible all I can do is provide guidance but I can't reliably solve this for you.

Regarding the code snippet I posted above, as mentioned previously, I posted it as a starting point and you should have no expectation of it working as is. FWIW, having a closer look I notice this in my library:

_eeg[j] = ((int)_payloadData[++i] << 16) | ((int)_payloadData[++i] << 8) | (int)_payloadData[++i];

while the code you've posted does this:

Hilf =  ((long)ReadOneByte() * 256 + (long)ReadOneByte());
if (Hilf > 32767)
          Hilf -= (long)(65535);

This leads me to believe the Data[] list of 512 values is in a different range compared to how the Mindwave library returns via .eeg() which means the rest of the code won't work as expected (since the condition won't match).

From a pragmatic point of view, for a faster solution I would consider one of the following options:

1. If you can use a PC with a bluetooth module simply use the NeuroSky Windows Developer Tools since they already provide support for Blink and Attention (not to mention a sample cpp app you can compile with Visual Studio and tweak to use for your project)
2. If your existing code, messy as it is works you can try reading the attention value in there, either looking at Mindwave.cpp in this repo or at mindwave_mobile_arduino_sample_code.docx which is more similar to the code you use.

For option 2 you will need to get familiar with ThinkGear Packets(page 9 onwards) and a bit of working with bytes (using hexadecimal, bits/bytes, bitwise operators (&(AND), bit shift left (<<), right(>>), XOR (^), OR (|), NOT(|) ). This is very useful CS knowledge regardless of the project and I'd say programming language, but also a great introduction to serial communications and how to put together a communication protocol (with a header, checksum and data payload). It may seem intimidating at first, but patiently going one step a time will get you very far (e.g. play with bitwise operators alone in a separate sketch first to get the hang of them, make a copy of the working and start commenting code until the start of the serial comms (e.g. if (ReadOneByte() == 170 && ReadOneByte() == 170) which is the same as if(ReadOneByte() == 170) {if(ReadOneByte() == 170) {...),) then adding one element at the time following the protocol to get to where the attention value is parsed from the payload (after checksum is validated): attention = payloadData[i];)

[mehaghiles]

@orgicus I'll try to take a look at all these documents and find a solution. I have already trying to work with the Mindwave.cpp code you gave me that i have foun on the Neurosky official website and I'm still trying to arrange the variables so that they are the same in both programs. I find some difficulties because i don't know exactly the role of each variable and how it works.
Anyway I think we should work together to add this bliking eye detection to your library so just contact me when you have enough free time. Meanwhile, I'll be greatful if you could just guide me to find a way to merge the blinking eye code and the mindwave.cpp code from the neurosky repo.

[mehaghiles]

@orgicus Here is the original blinking eye detection code wwith all the comments. It will certainly help understanding the code more.
`void loop()
{
long Hilf = 0;
// The Data are written in a drum with the circumference of 512 values (index i), time is running with increasing i
// i is the index of the current time, decreasing positions starting from i in the Data array means going back in the past
if (i >= 512)
i = 0;

if (ReadOneByte() == 170 && ReadOneByte() == 170)// this is the beginning of each new sequence
{
payloadLength = ReadOneByte();
if (payloadLength == 4)// the raw data have a payloadlength of 4, the esense values have another payloadlength < 170 bytes
{
if (ReadOneByte() == 128 && ReadOneByte() == 2)// the sequence announcing raw data is 170 170 4 128 2
{

    Hilf =  ((long)ReadOneByte() * 256 + (long)ReadOneByte());// read the most significant two bytes and form a signed number of it
    if (Hilf > 32767)
      Hilf -= (long)(65535);
    Data[i] = (int)Hilf;

    ReadOneByte(); // read the third byte, without taking notice of it
    // PiekP is a gliding sum over 50 values of Data 71 values of Data in the past, 71 values are reserved for the minus peak PiekM
    PiekP += Data[(512 + i - 71) % 512];
    PiekP -= Data[(512 + i - 50 - 71) % 512];
    // Test, if PiekP exceeds a certain value and the youngest value of PiekP is negative and it has no huge values
    if ((PiekP > 3000) && (Data[(512 + i  - 70) % 512] < 0) && (PiekP < 13000))
    { // The next eye blink detection is enabled only after a certain elapse time
      if (millis() - piekTime > 100) //time
      { PiekM = 0;
        // After detecting a positive peak PiekP the following 70 values are summed up and tested, if more negative than a certain value
        for (int j = 1; j <= 70; j++)
          PiekM +=  (int)(Data[(512 + i  + j - 70) % 512]);

        //Sometimes big negative numbers appear, which are suppressed by a limit for the negative values, if they are to huge
        if (PiekM < -3000 && PiekM > -11000) {
          
          //Serial.println("I-Blink detected!");
          // n is the counter for the number of subsequent eye blinks within a certain time difference
          if ((millis() - piekTime) < 600)n++; else n = 1;
          //Serial.print(PiekP);
          //Serial.print("; ");
          
          //Serial.println(PiekM);
          //Serial.print("   n   ");
          //Serial.println(n);
          //Serial.print("   poorQuality    ");
          //Serial.println(poorQuality);
          if(poorQuality == 0)digitalWrite(6,HIGH);else digitalWrite(6,LOW);
          piekTime = millis();//piekTime is the time at which the eye blink has been detected
          piekDetected = true;// piekDetected is set true, when an eye blink has been detected
        }// end if PiekM (eyeblink detected)
      }//end elapse time
    }// end PiekP detect
    i++;// move one raw value forward
    // The Data Array can be printed out if DEBUGOUTPUT is set true
    if (DEBUGOUTPUT && i == 512)
    {
      for (int j = 0; j < 512; j++)
      {
        Serial.print(Data[j]);
        Serial.print(";");
      }
      Serial.println("");
    }// end debug output
  }// end if 128 and 2 found
}// end if payloadLength == 4
else if(payloadLength < 170){// this is the access to the esense values, here poor quality, which is announced by a byte with value 2
  for (int k = 1; k < payloadLength; k++)
  if(ReadOneByte()== 2)poorQuality = ReadOneByte();
}

}// end if 170 170 appeared
//print n after a certain elapse time, peakDetected has the purpose, that the printing happens only once after the last eye blink
if((millis()-piekTime)>700 && piekDetected == true)Serial.println(n),piekDetected = false;
}// end of loop`
blink_eyes.zip

[Tafhimbn]

@orgicus Thank you for your response. 

It took a while for me but i finally found how to detect the blinking eye. I'm working on a project since months and i need to use both of blink detection and the attention value, unfortunatly i have some isues using your library with the code that i have. When I try to use the command "mindwave.update" nothing happens.

I was wondering if you could help me solving this problem.

please, can you tell me how you detect eye blink?

[Tony-file]

@orgicus @Tafhimbn @mehaghiles I am starting a project that will use eye-blinks, and in my research on the Internet I found an Arduino sketch that uses orgicus' library and successfully detect eye-blinks. Here is the link: https://www.instructables.com/FPV-Virtual-Reality-Arduino-Controlled-Tracked-Rob/ . The code for NeuroSky is in Step 11 and is titled "Code for the Brainwave Controlled Robot:"
The code needed some changes before it would compile. All the "<" "p" ">" needed to be removed and the first statement needed to be replaced by the following:
#include <Mindwave.h>
Mindwave mindwave;
(The ultrasonic sensor is not in my project, and so I deleted statements related to it.)
I am modifying the code for my project and have run some tests. It does detect blinks but it needs more work to do it properly. Just one example is that the "blink to stop" code is executed right after the "blink to start" (regardless of the "delay" statement) because the code is still working on the same signal from the headset instead of the next signal. Some code needs to be inserted between the two "blink" parts, to process a signal, and then the "blink to stop" code will work properly. The code I inserted in my tests is a little awkward and maybe strange, but it worked okay.

[AK-Homberger]

Hi orgicus,

I forked the library and integrated the eye blink detection in the code:
https://github.com/AK-Homberger/Mindwave

Feel free to include the changes also in the main branch.

Regards,
Andreas

[orgicus]

Hi @AK-Homberger Great work!

It would amazing if you could contribute this feature.
Please use the Create pull request button.

Unfortunately today I won't time to fully review.
Overall the code good looks. I might merge the contents of my README with your README, crediting you of course.
Let me know if that sounds good to you.

Many thanks,
George