Finding "plateaus" in data points

I'm thinking of a "running error"
If some tolerance of "plateauness" (ie. being in the same level -ness) is not crossed between subsequent amount of n points, then I'd consider the points which passed the test as a plateau.

-Nik

[edited by - Nik02 on February 10, 2004 2:46:47 PM]

The trick is to find the derivative of the function. It''s okay if it''s just the deltas between the data points.

Then, look for long areas of almost-zero. Those are your plateaus.

This is pretty much the same thing as Nik''s saying, just phrased differently.

---

"Sneftel is correct, if rather vulgar." --Flarelocke

quote:

---

Original post by Sneftel
The trick is to find the derivative of the function. It''s okay if it''s just the deltas between the data points.

Then, look for long areas of almost-zero. Those are your plateaus.

This is pretty much the same thing as Nik''s saying, just phrased differently.

---

"Sneftel is correct, if rather vulgar." --Flarelocke

---

I was looking for the word "delta", I''m not very good with English math terms

-Nik

Problem is, this isn''t a straight-up mathematical function: it''s output data from a torque measurement machine.

Yes, but you don't need to do anything else except to compare differences between some number of adjacent data points.
Noise is a function too - not necessarily rational or continuous one - but that doesn't matter here!

-Nik


[edited by - Nik02 on February 11, 2004 10:22:50 AM]

quote:

---

Original post by Nik02
I was looking for the word "delta", I''m not very good with English math terms

-Nik

---

It''s not and English maths term .

Δt ring a bell?

---

"Finishing in second place, simply means you are the first loser." - PouyaCat

quote:

---

Original post by JNewt

   /\
  /  \   ___
 /    \_/   \
/            \
 

As you can see, it has 2 peaks; I''m interested in the 2nd, specifically, identifying where the plateau starts and ends. The illustration above is macro-view, the real graph is pretty rough.

---

You might run a low-pass filter on the data to smooth it out a bit so that it more closely resembles the above. Then, as mentioned before, check the gradients and the derivatives.

You might also look into Sobel and Laplacian operators as they are designed for this sort of thing (finding changes of gradient over discrete data). What you''re doing is basically what an image processing package would call edge-detection. I believe the Sobel operators detect any non-zero gradient whereas the Laplacian one detects a change in gradient.

[ MSVC Fixes | STL Docs | SDL | Game AI | Sockets | C++ Faq Lite | Boost
Asking Questions | Organising code files | My stuff | Tiny XML | STLPort]

quote:

---

Original post by Crispy

quote:

---

Original post by Nik02
I was looking for the word "delta", I''m not very good with English math terms

-Nik

---

It''s not and English maths term .

Δt ring a bell?

---

In Finland, they teach the concept as "siirtymä" - not quite same sounding as "delta", is it?

But yes, I know the concept very well

kind rgds,
Nik

quote:

---

Original post by Nik02
In Finland, they teach the concept as "siirtymä" - not quite same sounding as "delta", is it?

But yes, I know the concept very well

---

Well, you Finns are weird (in a good way ) - here in Estonia we just use the Greek alphabet and everything works just fine - no ambiguity, portable language-wise and it even sounds cool.

---

"Finishing in second place, simply means you are the first loser." - PouyaCat

We use the greek alphabet too, in written formulas, but in my school teachers used to call it "siirtymä"
I guess it depends on a teacher, though.

Sorry for being off-topic, by the way!

-Nik

Let me bounce a couple of ideas off you.

1) My calculus teacher suggested smoothing the graph until it looked like my illustration, then running a simple slope analysis. OK, but will sacrifice accuracy and speed.

2) I think that I can move along the data points one at a time, grabbing the ten (or whatever) and finding their average. Then I''ll subtract the value of the lowest data point. The closer the average is to zero, the flatter the slope is. If this value is within a certain tolerance level (determined by trial and error) then I''ll assume that the plateau has started.

What do you think? Thanks for all the feedback.

If you''re dealing with a calculus teacher, then they should be able to explain how derivatives are the accepted answer to this problem, rather than ad-hoc assumptions about averaging values, etc. But yes, the first step should be a low-pass filter which smooths out the noise (high frequencies in the data).

Your second idea might work, but it''s a bit of a hack.

[ MSVC Fixes | STL Docs | SDL | Game AI | Sockets | C++ Faq Lite | Boost
Asking Questions | Organising code files | My stuff | Tiny XML | STLPort]

Kylotan : "derivatives are the accepted answer to this problem".
Hmmm, perhaps you could demonstrate the usefulness of derivatives in this problem?

[edited by - JNewt on February 12, 2004 8:50:44 AM]

Here is a definition of derivative.

Essentially, the derivative of a function is the rate of change of a given function (and delta is the rate of change at given point of the function).
Incidentally, the rate of change across some data points is what you want to check for when scanning for plateaus in the data

-Nik

[edited by - Nik02 on February 12, 2004 1:28:30 PM]

quote:

---

Original post by Nik02
Here is a definition of derivative.

Essentially, the derivative of a function is the rate of change of a given function (and delta is the rate of change at given point of the function).
Incidentally, the rate of change across some data points is what you want to check for when scanning for plateaus in the data

-Nik

[edited by - Nik02 on February 12, 2004 1:28:30 PM]

---

Derivatives would work amazingly well, but he doesn't have a mathematical function, he just has a bunch of sampled data. Without the formula, start at end and go down the list of data till you find a group that are all within a given "tolerance" away from one another, and make sure this check of tolerance passes over "x" points, x = however big of a plateau you want.


[edited by - stonicus on February 12, 2004 4:02:36 PM]

Couldn''t you set the points on an (x,y) graph and see the x-range of points that are close to the plateau?

Real data is always ugly.

If the plateau is the distinguishing feature, I would search for that first. Is there a "normal" width for the plateau? If so, I would do the following:

1) compute a delta vector [8^)
d = fabs(f[i+1] - f[i]);

2) compute a running sum vector over "width" samples
sum_delta += d[i+width/2] - d[i-width/2];
sd[i] = sum_delta;

3) find a minimum sd, call that the center of the plateau.

4) find the average plateau height

5) find where the data drops below a threshold (e.g. 90% of average height) by starting in the center and working your way left and right.

BTW, I think smoothing is not a bad idea, look into the precision/accuracy of the sensor used to gather the data and use that as a measure of how much you should smooth. Also, if you can eliminate some data (i.e. the left peak) this will run faster.

good luck,
lonesock

Piranha are people too.

Sfpiano: "Couldn''t you set the points on an (x,y) graph and see the x-range of points that are close to the plateau?"
Yep, but my program needs to process potentially hundreds of different sets of data.

Nik02: I know what derivatives are (nice explanation though) and have used them (instantaneous velocity, tangent slopes, etc.). I just wasn''t thinking of my problem in the proper mindset to intially understand the derivative suggestion. However, I think my method is already making use of such concepts, since it looks for places where the overall rate change is low.

Stonicus: Your suggestion was basically my 2nd idea (as described in an earlier post) right?

lonesock: Your suggestion was also very similar to what I''m working on implementing right now (just a bit more technically described). For speed I''m finding what I call the Max Peak and searching from there; the plateau always follows after this particular feature.

Thanks for the suggestions, all. Half of my problem is that I''m having to write all this in Excel''s VB macro language (gaaaahh).