GIS data storage

Aw, come on, I thought there are at least a few people here that knew more than just code monkeying. I mean, I'm perfectly capable of doing it myself, I just thought that it would be an interesting discussion, even for people that don't know much about GIS. In fact, I think that's what the field needs right now, is a healthy dose of diverse opinions.

There are several data structures for storing two dimensional data (points and lines) with efficient support for windowing queries (give me all objects withing the given region). I think it would make the most sense to have an RDMBS for connecting information to entities, while their location is stored in separate data structure. Or maybe that data structure is only used for lookups, like an index. But perhaps traditional databases can also solve this problem fast enough. giving a query with two where constraints can be seen as a winding query on a two dimensional dataset, so the problem an RDBMS solves is essentially the same.

How is the relational data stored now? What relations are actually being stored about the data? (right off the bat I'd agree that a tradition relational database is not the best medium for this kind of data - at the least you'd want to establish a layer above the database, with the database storing an optimized relational set of information generated from the above layer)

i have been interested in getting into the GIS realm so i have a few questions? what is it written in? yeah i know i can google or wiki but when someone is editing software for it or whatever then what do you use? or.. how does GIS do its drawings of vectors, points and lines? like i guess i am a little confused because i see a lot of places where they need C++ programmers for the support of GIS and i guess i am wondering how all that ties in together. thanks in advance capt

Which ArcGIS product are you using?

Unfortunately (or fortunately), I was using ArcGIS Engine, and had to load up several terrabytes of information, but it was a standalone map.

Something sounds terribly wrong with what you were doing, because the actual database we ended up using wasn't all that much bigger than the core data files.

Granted, we ended up loading the data in batches because the machines just didn't have enough space, nor memory to hold it all.

Wow, a little bit of interest now, cool. Long post commencing.

Quote:

Original post by twanvl There are several data structures for storing two dimensional data (points and lines) with efficient support for windowing queries (give me all objects withing the given region). I think it would make the most sense to have an RDMBS for connecting information to entities, while their location is stored in separate data structure. Or maybe that data structure is only used for lookups, like an index. But perhaps traditional databases can also solve this problem fast enough. giving a query with two where constraints can be seen as a winding query on a two dimensional dataset, so the problem an RDBMS solves is essentially the same.

yes, that is essentially what is done now with the RDBMS. It's a little more abstract than that, but that's basically it. However, I'm not convinced that it is good as it can be. For one, because of how GIS databases are setup, this isn't as simple as "select * from table where minX < x and x < maxX and minY < y and y < maxY". If it were that simple, I would almost be willing to accept the overhead.

Quote:

Original post by Michalson How is the relational data stored now? What relations are actually being stored about the data? (right off the bat I'd agree that a tradition relational database is not the best medium for this kind of data - at the least you'd want to establish a layer above the database, with the database storing an optimized relational set of information generated from the above layer)

usually, the relations are between geometry features and information about the feature. Geometry is usually stored in one table and metadata in another. They are usually linked by some sort of numerical ID number.

The point data itself is stored in a user type in another user type in the data base. There is a "Point" type and a "Collection" type. Points are stored in the Collection (these aren't the actual names, and depending on the geometry type they might be different as well), and the Collection is what is stored in the database. Yes, this means you essentially have a multi-valued column. Now, I studied database normal forms just like every good, little computer scientist, so obviously this makes me grit my teeth.

There is a layer on top of the database. In the case of Oracle Spatial, it's pretty small, offering only basic functionallity in spatial querying. In the case of ArcGIS Server, it's extremely heavy and bloated. Oracle Spatial isn't half bad, actually. Perhaps my problem is just ESRI products, idk.

Quote:

Original post by OpenGL_Guru i have been interested in getting into the GIS realm so i have a few questions? what is it written in? yeah i know i can google or wiki but when someone is editing software for it or whatever then what do you use? or.. how does GIS do its drawings of vectors, points and lines? like i guess i am a little confused because i see a lot of places where they need C++ programmers for the support of GIS and i guess i am wondering how all that ties in together. thanks in advance capt

For a development language, it depends on what you are doing. As I said before, the field is mostly GIS analysts that learned to program (though it's getting better). GIS right now is very web-application-centric, so in that realm it basically boils down to Java or ASP.NET. There is a significant desktop app development side still, and it was dominated by VB6 before, now Java. This basically involves writing a user interface over ESRI products, though there are other options available, and supporting all of them can make life interesting. And of course, a 3rd party library never fits the exact task that one needs it to do, so one usually ends up writing a significant portion of proprietary code to fill in the gaps. This is the main bulk of GIS work.

If you were to work for a company like ESRI, developing spatial servers and map renderers, you'd probably use C++. However, I suspect ESRI uses MS VC6, as I know for a fact that one of their sub-development teams that develops a specific plugin DO use it. Indeed, a memory leak in the MS version of the C core runtime library from VC6 was causing a huge bug for us. I only knew about the leak because a friend of mine had tracked it down in one of his research projects before. It's a total pain in the ass one, too, it takes about 6 hours of heavy processing before it becomes a problem. It took me months to convince the dev team that that was the problem, but that's a different situation altogether.

Map rendering is done in software, near as I can tell. GIS analysts go on and on about how fast it is, which makes me shudder to think of what they consider slow. I've done some preliminary design and testing on making an OpenGL based map renderer, and results have been very promising.

I have some links to some stuff that might interest you in my wiki. Also, here is a link to a number of OSS GIS Projects.

One OSS mapping app that I have had some experience with is BBN OpenMap. It's a Java desktop application, but I hacked a chunk out of it to implement a specific layer renderer for a Java web app. It's an... interesting... program, to say the least. It's one of the best OSS GIS applications out there... unfortunately that doesn't mean it's very good. I find the UI to be confusing, like features are hidden away, the application itself to be unstable (though this isn't too different from commercial apps, the most similar being ESRI ArcMap), and the code to suffer from typical Java style "Enterprisy Abstractionism" (Hey, I think I'll call that JaSEA, as it like Asia with a J on the front).

Quote:

Original post by Nytegard Which ArcGIS product are you using? Unfortunately (or fortunately), I was using ArcGIS Engine, and had to load up several terrabytes of information, but it was a standalone map. Something sounds terribly wrong with what you were doing, because the actual database we ended up using wasn't all that much bigger than the core data files. Granted, we ended up loading the data in batches because the machines just didn't have enough space, nor memory to hold it all.

Well, the largest one that I have in mind is IRRIS. We develop it as an ASP.NET web app for use by the Army to track shipments of explosives throughout the world. Part of this task includes a lot of report generation, toxic gas cloud simulations, route planning, and even heuristic-based alerting to indicate when a shipment of interest comes near an important piece of infrastructure (i.e. a shipment of 500kg bombs drives over the Brooklyn Bridge. What happens if that truck breaks down? Or a hostile party detonates a small bomb next to it, expecting tertiary explosions?).

Mind you, that's just it's original purpose, it has suffered significantly from feature bloat. It probably loads just about every data set one could get for a GIS application. At this point, we can drop it into just about any task we need to. We don't, because A) it's often too hard to remove the functionallity that a client DOESN'T need and doesn't have the security clearance to even SEE, B) a specific client may not have the resources to run the thing (it's a beast), and/or C) they might desire a Java solution. Barring A, B, and C, it's usually just a matter of assigning two developers to deploy the app, write new features that this specific customer wants (which end up getting rolled back into the main app branch), having one graphic designer reskin the app, and that's about it. Deployment is probably the hardest task, it can be quite involved at times.

Just out of interest (and feel free to ignore me if the question is too off-topic), what is the most impressive GIS visualization you've seen? Visual Nature Studio blows my socks off... have you come across anything better?

To add something* to the topic, IMHO problems related to GIS storage are very similar to problems related to scene graphs. Ideally, there'd be one storage solution and one data representation, but that isn't the case - GIS have multiple data representations (e.g. raster, vector and object -based), each having different semantics and a different range of meaningful operations. Kyle Wilson makes similar observations about the variety of storage and representation systems available to scene graph design.

I must admit that most of your post goes over my head - my GIS experience is limited to supporting GRASS in the GIS labs at my university, plus my supervisor is the GIS lecturer in my department - but I can agree that a relational database isn't likely to be the most optimal storage solution for a "multi-paradigm" data set.

*most likely unintelligible and/or unintelligent...

It's a little surprising that they aren't using spatial data structures. I assume that it has something to do with kD-trees and the like not being suitable for searching extremely large datasets, i.e., datasets where there are a large number of tree levels to pull off the disk.

Is there any common language for native formulation of spatial queries? That is, given a query like "find all the gas stations within five miles that aren't in Pennsylvania", is there an intermediate representation which looks more like "SELECT * from gas_stations WHERE (location WITHIN 5 OF myLoc) AND (location NOT INSIDE (SELECT area FROM states WHERE abbrev='PA'))" than a big hairy thing with euclidean distances and unions and suchlike stuck into the SQL?

Quote:

Original post by ajones Just out of interest (and feel free to ignore me if the question is too off-topic), what is the most impressive GIS visualization you've seen? Visual Nature Studio blows my socks off... have you come across anything better? To add something* to the topic, IMHO problems related to GIS storage are very similar to problems related to scene graphs. Ideally, there'd be one storage solution and one data representation, but that isn't the case - GIS have multiple data representations (e.g. raster, vector and object -based), each having different semantics and a different range of meaningful operations. Kyle Wilson makes similar observations about the variety of storage and representation systems available to scene graph design. I must admit that most of your post goes over my head - my GIS experience is limited to supporting GRASS in the GIS labs at my university, plus my supervisor is the GIS lecturer in my department - but I can agree that a relational database isn't likely to be the most optimal storage solution for a "multi-paradigm" data set. *most likely unintelligible and/or unintelligent...

I would have to say NASA WorldWind is the most impressive that I've seen thusfar. I've seen other things similar to VNS2 before, and while they look pretty in screenshots, they are horrible in live motion. Plus, most 3D terrain rendering applications like that aren't actually that useful beyond the 1 or 2 eye candy features they have.

Quote:

Original post by Sneftel It's a little surprising that they aren't using spatial data structures. I assume that it has something to do with kD-trees and the like not being suitable for searching extremely large datasets, i.e., datasets where there are a large number of tree levels to pull off the disk. Is there any common language for native formulation of spatial queries? That is, given a query like "find all the gas stations within five miles that aren't in Pennsylvania", is there an intermediate representation which looks more like "SELECT * from gas_stations WHERE (location WITHIN 5 OF myLoc) AND (location NOT INSIDE (SELECT area FROM states WHERE abbrev='PA'))" than a big hairy thing with euclidean distances and unions and suchlike stuck into the SQL?

Like I said, usually data is stored in a custom user type. You retrieve the points of interest by creating different filters that are used in the WHERE clause. Some of it is standard SQL, some of it is proprietary add on. It's all very fractured and definitely not intuitive. This page has a few examples of some simple queries at the bottom. They are much simpler in syntax than anything I've had to do for real work. I usually have to interface through ArcSDE. SDE is basically a layer on top of an RDBMS to make it usable with GIS tasks. Later today I'll try to post one of my examples.

I'm glad to see that I'm not the only one complaining. The author includes a sample query for an "ideal" spatial dbms, that I agree would be great. This is similar to how Oracle Spatial works, but again, it's not quite there and it's limited in its feature set (or so I'm told by our GIS analysts). There are some excellent comments after the article as well.

It appears that there is some work being done in this direction, which makes me glad, but it's still not here, because we're all still using SDE.

[Edited by - capn_midnight on September 21, 2006 10:18:28 AM]

did you stay with VC6 after you found the bug or were able to move into .NET and use C++ in that environment?

if you work with ESRI stuff like you mentioned using C++, what would you use C++ for? why not just java? what does C++ do to make it more feasible?

Quote:

Original post by OpenGL_Guru did you stay with VC6 after you found the bug or were able to move into .NET and use C++ in that environment? if you work with ESRI stuff like you mentioned using C++, what would you use C++ for? why not just java? what does C++ do to make it more feasible?

Oh, I don't use C++, I was using C# at the time. We were using a plugin called GlobeXplorer for the ArcGIS mapping server. ArcGIS is an ESRI product, and GlobeXplorer is written by an ESRI subsidiary (supposedly they had access to the mapping engine's source code). I had seen the source code to GlobeXplorer, and I could see it was a VC6 project.

You can interface with ESRI products in a variety of languages, C, VB6, Java, .NET in general. You can also setup web services with it that can be access with any language that cares to implement a web services consumer library.

Quote:

Original post by capn_midnight Quote: Original post by OpenGL_Guru did you stay with VC6 after you found the bug or were able to move into .NET and use C++ in that environment? if you work with ESRI stuff like you mentioned using C++, what would you use C++ for? why not just java? what does C++ do to make it more feasible? Oh, I don't use C++, I was using C# at the time. We were using a plugin called GlobeXplorer for the ArcGIS mapping server. ArcGIS is an ESRI product, and GlobeXplorer is written by an ESRI subsidiary (supposedly they had access to the mapping engine's source code). I had seen the source code to GlobeXplorer, and I could see it was a VC6 project. You can interface with ESRI products in a variety of languages, C, VB6, Java, .NET in general. You can also setup web services with it that can be access with any language that cares to implement a web services consumer library.

so you can start a new C++ project and actually change the behavior of how an ESRI product reacts such as changing the way it renders lines, vectors, maps etc or maybe you can add your own stuff directly to the application? i just want to make sure i understand you and if you can do that that is quite handy.

Quote:

Original post by OpenGL_Guru Quote: Original post by capn_midnight Quote: Original post by OpenGL_Guru did you stay with VC6 after you found the bug or were able to move into .NET and use C++ in that environment? if you work with ESRI stuff like you mentioned using C++, what would you use C++ for? why not just java? what does C++ do to make it more feasible? Oh, I don't use C++, I was using C# at the time. We were using a plugin called GlobeXplorer for the ArcGIS mapping server. ArcGIS is an ESRI product, and GlobeXplorer is written by an ESRI subsidiary (supposedly they had access to the mapping engine's source code). I had seen the source code to GlobeXplorer, and I could see it was a VC6 project. You can interface with ESRI products in a variety of languages, C, VB6, Java, .NET in general. You can also setup web services with it that can be access with any language that cares to implement a web services consumer library. so you can start a new C++ project and actually change the behavior of how an ESRI product reacts such as changing the way it renders lines, vectors, maps etc or maybe you can add your own stuff directly to the application? i just want to make sure i understand you and if you can do that that is quite handy.

Not really. I misspoke about GlobeXplorer. It wasn't really a plugin as much as it was a patch. I don't understand the reason why they designed it as a patch on ArcGIS. Basically, they took the ArcGIS source code, hacked three or four DLLs, and then released the DLLs. GlobeX served up some hi-res raster imagery. I don't understand why they did it that way, it's fully possible to add custom raster imagery to a map without any code at all, it can all be done in the database and config files. Let's just say that this kind of stupid stuff is typical of the field.

However, you CAN use C++ to write applications that use ESRI products. It's basically a collection of libraries and services that you can interface with for the purpose of RAD GIS application development.

Hope this is too obvious but testing against the squared distance should be much faster.

i.e. to find all the locations with 5 units of a point

SELECT locations.*
FROM locations
WHERE SQUARE(locations.x - pointx) + SQUARE(locations.y - pointy) < 25