Johannes Kebeck has an interesting post showing how to get the benefits of the MapPoint Web Service method, FindNearRoute, through the use of SQL Server 2008. He uses a newly-added feature of the Virtual Earth API that allows developers to gain access to a complete returned route-geometry when performing a route. This new feature, however, does require that the mapping application use customer identification (which, in turn, requires that you sign up for a Virtual Earth platform developer account).

If you haven’t checked out Johannes’ blog lately, you might want to give it a go now. He has written a whole slew of Virtual Earth and SQL Server 2008 posts lately, and there’s a wealth of good information there.

We’ve already started using SQL Server 2008 in some of our development applications, and are planning on taking advantage of ArcSDE 9.3′s ability to utilize SQL Server 2008′s spatial data types. We’re hoping that by doing this, we’ll be able to offer access to our geometries through both traditional GIS tools (ArcGIS Desktop for our shop) and web services (ADO.Net Data Services returning JSON and XML and other services built on top of WCF). As we’re already using ArcGIS Server 9.3′s REST capabilities from within our JavaScript/Virtual Earth applications, if we can “close the circle” by tying SQL Server 2008 and Virtual Earth together in a robust and meaningful way, we may just be able to hit the proverbial bulls-eye. We’ve already written quite a bit of code that brings these technologies together, but are still deciding on the best overall approach.

One thing that is still unclear to me is if we’re going to utilize ESRI’s tools heavily or try to avoid them as much as possible and stick with all of the non-ESRI .NET technologies. In a lot of ways, we’re already going the latter route (saying ArcGIS Server 9.2 still leaves a bad taste in my mouth), but I’ve been impressed with ArcGIS Server 9.3 so far, so I’m withholding judgement until we see how the new set of technologies perform in a couple of months.

And from the Microsoft camp, I expect that more direct integration with SQL Server 2008 will be coming soon to the Virtual Earth API. I haven’t, however, heard of any solid dates yet. Any integration will be much-welcomed, as it will save us precious development time.

Whichever route we decide to take, it’s pretty obvious that now is a great time to be working in web mapping. Especially if you’re able to take advantage of all the integration points available through .NET.

Those of you who use Microsoft SQL Server will be interested to hear that the SQL Server team has just released the November Community Technology Preview of SQL Server 2008 “Katmai”. Of special note for us GIS types is that this is the first publicly available release that includes support for the new spatial data types.

The last couple of weeks have seen a slew of news about this upcoming release, including some pretty interesting demonstrations of the new spatial data types. A good blog to keep your eye on is Isaac’s MSDN blog; he recently linked to a webcast that he gave on the 1st of November, titled “Introduction to Spatial in SQL Server”. This is a good place to start if you’re just being introduced to the new capabilities.

Here’s a link to the download.

I’ll be playing around with Virtual Earth and this release quite a bit over the next month and will post about my experiences here.

This past week, ESRI released several new white papers. The one that I’m most interested in is “An Overview of Distributing Data with Geodatabases“. This is the white paper that was promised during the “Managing Distributed Data with Geodatabase Replication” session at the ESRI 2007 UC.

The first several pages are full of the now obligatory replication/synchronization diagrams which, while maybe helpful to some, I’ve seen at least a hundred times now. The post then, however, goes into the guts of the different techniques, including geodatabase replication, DBMS replication, and simple copying and loading of data. Here’s a quick summary of each technique:

• Geodatabase replication replicates data from a source (parent) geodatabase to a destination (child) geodatabase and creates the infrastructure to allow for tracking and synchronization of changes. Geodatabase replication is fairly powerful; it allows parameters to be defined that restrict the data that are replicated. At 9.2, geodatabase replication can only occur on SDE geodatabases, but at 9.3 SDE geodatabases will be able replicate one-way with a file or personal geodatabase. There are different types of replicas, including check-out/check-in, one-way, and two-way.

Developers can work with replicas through the GeoDatabaseDistributed ArcObjects library components. If you just need coarse-grained access to the geodatabase, you can use the GeodataServer object model. If you need more fine-grained access, however, you will want to use some of the older ArcObjects object models which allow you more capabilities but restricts you to working with geodatabases that live on your Local Area Network.

Geodatabase synchronization can be automated using the synchronize changes geoprocessing tool (create a Python script, call it from a script (.bat or otherwise), and schedule it using Windows scheduler).

In cases where large amounts of data are stored in a geodatabase and need to be moved to a remote office and synchronized, it may be best to detach the database from the parent, send it to the child (mail it, etc.), and then reattach it. You can then use the ‘register with existing data’ tool to create the replica.

• DBMS replication is good for replicating an entire geodatabase to a read-only geodatabase in a connected system. Therefore, this is a good option for load balancing and fail over. The major limitation is the inability to choose what gets replicated.

To setup and use DBMS replication, you must first understand how a geodatabase interacts with its underlying database.

• Data copying and loading is the “low-tech” way of setting up replication between multiple geodatabases. There are multiple geoprocessing tools available to help automate and schedule this process. Using this method, however, there is not a great way to verify data integrity during the replication. You can write your own code to avoid redundancy, but this is a lot of work compared to just using one of the other replication techniques.

This white paper didn’t go as in-depth as I was hoping for. It does, however, tie the concepts together, and it contains links to other helpful resources. We’ve been using one-way replication successfully for a while now, and are looking to setup several two-way replicas with some remote offices here shortly. I’m sure you’ll hear back from me about our experiences.

Here are a couple of other resources to help you learn more about distributed geodatabases. The technical brief is especially useful:

• Link to the “An Overview of Distributing Data with Geodatabases” white paper
• ESRI’s “Geodatabase Replication: An Overview” Podcast (link to transcript)
• ESRI Systems Integration Technical Brief: “ArcSDE Geodatabase Replication Overview“

I’ve been experimenting quite a bit recently with MapDotNet Server 2007 (from hereon, referred to as MDNS), and I’ll say that I’m pretty impressed. Building maps in MDNS is admittedly more difficult than simply publishing an .mxd with ArcGIS Server – MDNS uses MapServer’s map file format, but the power of gaining access to Virtual Earth base data far outweighs the trouble that it takes to build the application and the performance is killer when compared to the alternatives. One thing I’ve noticed, though, is that the documentation/support site for MDNS still needs a lot of work. It’s obvious that ISC has put some work into their help in the recent past, but there are still lots of topics that need to be covered in much more detail.

One of the topics that is only briefly covered in the support portal wiki is integrating ArcSDE with an MDNS application. The steps are briefly outlined in the “ArcSDE Integration” wiki entry, but there really needs to be a comprehensive step-by-step walkthrough for a topic as important as this. If it were a true wiki, I would be able to edit the page and update the documentation for others to use, but I have yet to find a way to contribute. So I’m going to put up my little walkthrough here (as much for myself as for anyone else) and hope that it either gets integrated into the support site or that ISC updates their documentation soon.

There are three steps to the process, first you need to download the “libmap.dll.sde92″ dll (it wasn’t included with the MDNS 2007 6.0 release for some reason), second you need to enable ArcSDE connections via MapDotNet Server 2007, and last you need to connect your specific application to an SDE geodatabase. Make sure that you perform all of the steps on both your development machine and your web server.

1. Download the “libmap.dll.sde92″ dll:
   1. Click on the following link to download a .zip with the dll in it.
   2. Unzip the dll and put it in the “\IGNORED_LIBMAP_DLLS” directory, which is located at “C:\Program Files\MapDotNet Server\v6.0\MapDotNetServer Web Service\bin\IGNORED_LIBMAP_DLLS” in a default install.
2. Enable ArcSDE Connections via MapDotNet Server 2007:
   1. Install the ArcSDE 9.2 C SDK:
      1. The C SDK is included with your ArcSDE 9.2 install media.
      2. Initiate the install by either inserting the disk and using the Autorun installation GUI or browsing to the “\ArcSDE\windows\ArcSdeSDK” directory and double-clicking “setup.exe”.
      3. Choose “ArcSDE C SDK” from the installation window and let it run to completion.
      4. Add a Path variable to point to the ArcSDE\bin directory:
         1. In Windows, right-click on “My Computer” and select “Properties” from the context menu.
         2. Click on the “Advanced” tab and then the “Environment Variables” button near the bottom of the window.
         3. In the “System variables” section, scroll down to the “Path” variable and double-click it to open the “Edit System Variable” window.
         4. Go all the way to the end of the “Variable value:” field and add the following to the end: “;C:\ArcGIS\ArcSDE\bin” (without the quotations).
         5. Click “OK” three times to exit out of all the dialogs.
      5. Reset IIS by opening up a command line and running “iisreset”.
   2. Browse to the “\IGNORED_LIBMAP_DLLS” directory, which is – again – located at “C:\Program Files\MapDotNet Server\v6.0\MapDotNetServer Web Service\bin\IGNORED_LIBMAP_DLLS” in a default install, and copy and paste the “libmap.dll.sde91″ dll into the “\bin” directory.
   3. Stop the “World Wide Web Publishing” service, as it likely has a lock (via the aspnet_wp) on the “libmap.dll”:
      1. Right-click on “My Computer” and select “Manage”.
      2. Expand the “Services and Applications” console.
      3. Click on “Services”.
      4. Scroll down to the bottom, right-click on “World Wide Web Publishing”, and select “Stop” from the context menu.
   4. Cut and paste the “libmap.dll” dll from the “\bin” directory into the “\IGNORED_LIBMAP_DLLS” directory (just for safe-keeping).
   5. Rename the “libmap.dll.sde92″ dll that you placed in the “\bin” directory to “libmap.dll”.
   6. Restart the “World Wide Web Publishing” service:
      1. Back in “Computer Management” (see #3 above), restart the service by right-clicking on it and selecting “Start” from the context menu.
3. Connect Your Application to an SDE Geodatabase:
   1. Open your application’s .map file using either the “MapFile Generator” or your preferred text editor.
   2. Add a new layer to your .mapfile, using the following connection information (replace the red with your data source information):

CONNECTION “SERVER,port:PORTNUMBER,DATABASE,USERNAME,PASSWORD”
CONNECTIONTYPE SDE
DATA “DATABASE.USERNAME.NAMEOFFEATURECLASS,SHAPE,SDE.DEFAULT”
PROCESSING “CLOSE_CONNECTION=DEFER”

As many of you probably already know, ESRI’s upcoming ArcGIS 9.2 release will greatly enhance the ability of an organization to expand its enterprise GIS to multiple sites through the use of geodatabase replication and synchronization. This ability comes with the workgroup and enterprise versions of ArcSDE; it does not work with personal or file-based geodatabases.

Data have to be registered with the geodatabase and versioned before they are replicated, and a globally unique identifier (GUID) has to be added to each table record to aid in the tracking of multiple copies of the same tables (feature classes). Thankfully, there are ways to limit the data that are synchronized between geodatabases, allowing for a database administrator to control the amount of bandwidth used – and the data exchanged – by replicating and synchronizing only the data that are needed.

As far as versions go, they are replicated and synchronized with the geodatabase, as well. This is obviously necessary, but I can see it being both a good and a bad thing, depending on how you look at it. One thing is for sure, though: the addition of these features to the geodatabase, while adding a tremendously useful resource to the toolset of a geodatabase administrator, adds another dimension of complexity to the SDE world.

I am currently working on a project that will eventually replicate a geodatabase between multiple (10-50) sites, including the management and replication/synchronization of over 300 versions of multiple enterprise datasets from these sites. Up until the 9.2 release, this project wasn’t a possibility for us, as many of the sites that we are focusing on have iffy – at best – network connectivity. Also, the check-in/out capabilities that currently exist in the ESRI world have problems that make them an undesirable solution in many situations.

I plan on detailing the progress of our project as it progresses. I’m sure we’ll run into many pitfalls and (hopefully) some triumphs along the way.

Listening to Calexico – Garden Ruin…

I’ve been looking for a better way to design, diagram, and document my geodatabase designs.

Visio is my tool of choice for database design. It is easy to use and allows me to visualize and demonstrate the capabilities and limitations of my data model before actually creating the geodatabase. Using Visio, I can generate a UML schema and export it to XMI. From there, it is possible to generate all of the objects that exist in the data model into a geodatabase. There are other tools available, in addition to Visio. I haven’t yet experimented with them, though.

For diagramming geodatabases that weren’t designed in Visio, I use Geodatabase Diagrammer, which is available for download on ArcScripts. This sweet little utility allows you to export a diagram straight out of a geodatabase (with the newest release, it supports both personal and enterprise) into Visio. It was initially developed as an internal ESRI tool, but was released to the public to help in the documentation of geodatabases. One problem that the tool has, though, is that it uses the default ESRI layout for the diagrams – which probably isn’t that big of a deal for most. Another weakness that I see is that, like Geodatabase Designer 2, you have to export the entire geodatabase schema. This likely isn’t much of a problem with personal geodatabases (or maybe it is for some???), but it can be a pain to export an entire SDE geodatabase schema out and then pick through the Visio model to get just the elements that you want in your diagram. I usually have more than one “data model” in my enterprise geodatabases, and would like to be able to select the feature classes that are associated with a particular model and get only those relationships classes and related tables that are tied to the model.

For documentation of geodatabases, I use Richie Carmichael’s (ESRI) Geodatabase Designer 2, and it works well for what it was designed to do. It allows you to import and export Geodatabase XML schemas and documents all of the tables (spatial and otherwise), subtypes, domains, relationships, and networks that exist in a geodatabase in both HTML and XML. One shortcoming that the tool has is the fact that, like the geodatabase diagrammer, you cannot pick and choose “parts” of a geodatabase to export. On the positive side, though, this is obviously a valuable tool. It would be really valuable if I could figure out how to convert the XML into a Visio model. If only we could reverse engineer an SDE geodatabase!

Bottomline from what I’ve found is that if you didn’t use UML to initially design your geodatabase, there is no single tool or method that will allow you to easily diagram your model. If you, however, did design your initial data model in Visio, you can then use that model as a visual tool and generate your geodatabase from it. To document all of the contents of the model, use Geodatabase Designer 2.

I would love to hear about other methods of designing and documenting geodatabases. Any ideas?

Helpful Links:

• Building Geodatabases with CASE Tools
• Visio 2003 UML to XMI Export Tool

Listening to Neil Young – After the Gold Rush…

I’ve worked with PostgreSQL for a while now, but everytime I perform a new
install (usually with PostGIS added on soon after the initial install) I am amazed
at how simple it is and how smoothly it goes. In the past, I’ve worked on some
pretty complex database installs – especially complex, at least for me, when
installing to a Linux system – but it really only takes a few minutes to get
PostGIS up and running on a database on any
platform.

If you haven’t checked them out, you should go download them both now. And
if you want an open source desktop interface, there are a number of options that
plug directly into PostGIS, including Quantum GIS and uDig. With them you can easily connect to a PostGIS database,
and load (check out the Import Shapefiles to PostgreSQL plugin for Quantum
GIS) and edit data. On top of this, you can greatly extend Quantum GIS’
capabilities by using it as a graphical user interface for GRASS.

I agree with others that the major “gap” in the open source GIS arena is on
the client side of things. At this point, there isn’t an interface that rivals
the ArcGIS Desktop suite. The products that are available, though, do serve a
purpose. And if you’re willing to put the time into learning and integrating
them, they fill a huge niche that is left empty by ESRI.

As for the open source spatially-aware database, however, PostgreSQL and
PostGIS offer a viable – albeit not as powerful – alternative to the expensive
proprietary solutions. The two combined don’t work for every scenario, but they
do work for a surprising number of cases.

More to come on all of these solutions later…

A quick and easy way to prepare a SQL Server database for use in an ASP.NET application:

1. Browse to and execute aspnet_reqsql.exe, which is located at: C:\Windows\Microsoft.NET\Framework\VersionOf.NETFramework\.
2. Choose to “Configure SQL Server for application services” when prompted.
3. Specify the server, instance name, database name, and the username and password of a database user with the necessary permissions.
4. Complete the process by closing out the wizard.

Listening to Weezer – The Blue Album…

This may be obvious to most, but being new to ASP.NET – and relatively new to programming in general – I found it difficult to connect my ASP.NET application to a SQL Server instance. Looking at the available documentation, connecting should be easy. You simply create a connectionStrings element in your application’s web.config file and point it to the right server and database, right?

Well, yeah, but only after clearing out the default inherited connection settings. Out of the box, ASP.NET points to a SQL Server 2005 instance named, of course, SQLEXPRESS. If you don’t have a SQLEXPRESS instance running on your server, you have to either change the machine.config file located at

SystemDrive\WINDOWS\MICROSOFT.NET\FrameworkXX\vX.X.XXXXX\CONFIG

or clear out the inherited connection settings using the clear element and then add in your own connection settings to the web.config file in your web application. Here’s the code to clear out the inherited connection settings and connect your application to another – local – SQL Server database:

name=”LocalSqlServer”
connectionString=”data source=.\YourSQLServerInstance;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|YourDatabase.mdf;User Instance=true”
providerName=”System.Data.SqlClient”
/>

After you specify the correct connection settings, you should be able to use your SQL Server database to store your application’s data.