Read the Descriptions from all columns in MS Access database

From one of my developers. This VBA (couldn’t find an approach in .Net) will loop through all the tables and write out the description from all the fields.

 

TODO’s include grabbing descriptions for tables and then tweaking below to create ALTER statements so that the comments can be applied to a server RDBMS after migration from Access (since almost none of the migration utilities I’ve seen migrate this documentation from Access).

 

Option Explicit

[cc lang='vb' ]‘call readAllTables
Public Function readAllTables()

Dim DB As Database, tbl As TableDef, fld As DAO.Field

Dim RS As Recordset
Dim Table As String
Dim allDesc As String

Set DB = CurrentDb()

For Each tbl In DB.TableDefs
If Left$(tbl.Name, 4) <> “MSys” Then
‘Debug.Print “In Table ” & tbl.Name ‘& “      ” & tbl.DateCreated & “      ” & tbl.LastUpdated & “     ” & tbl.RecordCount
allDesc = allDesc & vbNewLine & “Table:” & tbl.Name
‘ optional code to print all the fields
On Error Resume Next
For Each fld In tbl.Fields
‘Debug.Print fld.Name
allDesc = allDesc & vbNewLine & fld.Name & “:” & fld.Properties(“Description”)
Next fld
End If
Next tbl

WriteToATextFile (allDesc)
End Function

Sub WriteToATextFile(ByVal outputStr)
‘first set a string which contains the path to the file you want to create.
‘this example creates one and stores it in the root directory
Dim MyFile As String
MyFile = “c:\” & “TableFieldsWithDesc.txt”
‘set and open file for output
Dim fnum As Integer
fnum = FreeFile()
Open MyFile For Output As fnum
‘write project info and then a blank line. Note the comma is required
Write #fnum, outputStr
Write #fnum,
Close #fnum
End Sub[/cc]

Handy database documenter/profiler for mysql, cont.

Some time ago I wrote down a little script to make a table from the MySQL information schema to describe your database. My eventual goal is to come close to reproducing a poor man’s database profiling script similar to this crude one ( http://www.ipcdesigns.com/data_profiling/), but perhaps less powerful and yet more elegant. I figure it’s going to take creating some procedures to loop through the chosen tables and columns.

 

Towards that end, I figure I need to take the contents of the handy view I made earlier and turn them into a table. Then if I execute some profiling queries, I can create tables from the results and join back to this summary table. So here is me persisting the view created earlier.You can do it this way:

[cc lang='sql' ]CREATE TABLE profiler_recs AS SELECT * FROM `v_field_table_data`;
ALTER TABLE `mysql`.`profiler_recs` ADD COLUMN `PROFILE_RECS_ID` BIGINT(20) NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY(`PROFILE_RECS_ID`); [/cc]

or here is the resulting DDL

[cc lang='sql' ]CREATE TABLE `profiler_recs` (
`PROFILE_RECS_ID` BIGINT(20) NOT NULL AUTO_INCREMENT,
`FIELD_NAME` VARCHAR(194) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`SCHEMA_NAME` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`DEFAULT_CHARACTER_SET_NAME` VARCHAR(32) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`DEFAULT_COLLATION_NAME` VARCHAR(32) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`SQL_PATH` VARCHAR(512) CHARACTER SET utf8 DEFAULT NULL,
`TABLE_CATALOG` VARCHAR(512) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`TABLE_SCHEMA` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`TABLE_TYPE` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`ENGINE` VARCHAR(64) CHARACTER SET utf8 DEFAULT NULL,
`VERSION` BIGINT(21) UNSIGNED DEFAULT NULL,
`ROW_FORMAT` VARCHAR(10) CHARACTER SET utf8 DEFAULT NULL,
`TABLE_ROWS` BIGINT(21) UNSIGNED DEFAULT NULL,
`AVG_ROW_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`DATA_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`MAX_DATA_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`INDEX_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`DATA_FREE` BIGINT(21) UNSIGNED DEFAULT NULL,
`AUTO_INCREMENT` BIGINT(21) UNSIGNED DEFAULT NULL,
`CREATE_TIME` DATETIME DEFAULT NULL,
`UPDATE_TIME` DATETIME DEFAULT NULL,
`CHECK_TIME` DATETIME DEFAULT NULL,
`TABLE_COLLATION` VARCHAR(32) CHARACTER SET utf8 DEFAULT NULL,
`CHECKSUM` BIGINT(21) UNSIGNED DEFAULT NULL,
`CREATE_OPTIONS` VARCHAR(255) CHARACTER SET utf8 DEFAULT NULL,
`TABLE_COMMENT` VARCHAR(2048) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`TABLE_NAME` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`COLUMN_NAME` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`ORDINAL_POSITION` BIGINT(21) UNSIGNED NOT NULL DEFAULT ’0′,
`COLUMN_DEFAULT` LONGTEXT CHARACTER SET utf8,
`IS_NULLABLE` VARCHAR(3) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`DATA_TYPE` VARCHAR(64) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`CHARACTER_MAXIMUM_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`CHARACTER_OCTET_LENGTH` BIGINT(21) UNSIGNED DEFAULT NULL,
`NUMERIC_PRECISION` BIGINT(21) UNSIGNED DEFAULT NULL,
`NUMERIC_SCALE` BIGINT(21) UNSIGNED DEFAULT NULL,
`CHARACTER_SET_NAME` VARCHAR(32) CHARACTER SET utf8 DEFAULT NULL,
`COLLATION_NAME` VARCHAR(32) CHARACTER SET utf8 DEFAULT NULL,
`COLUMN_TYPE` LONGTEXT CHARACTER SET utf8 NOT NULL,
`COLUMN_KEY` VARCHAR(3) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`EXTRA` VARCHAR(27) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`PRIVILEGES` VARCHAR(80) CHARACTER SET utf8 NOT NULL DEFAULT ”,
`COLUMN_COMMENT` VARCHAR(1024) CHARACTER SET utf8 NOT NULL DEFAULT ”,
PRIMARY KEY (`PROFILE_RECS_ID`)
) ENGINE=INNODB AUTO_INCREMENT=27399 DEFAULT CHARSET=latin1[/cc]

I think you could design a procedure(returnExtents) that would accept a schema_name, then loop through all tables and columns by selecting from the view or table we created earlier and store the results as follows

Accept: SCHEMA_NAME
Return: max_value, min_value, num_nulls, max_length, min_length for each record in the above table. Or one record for each column in the schema.

Ideally you would write the results into a table as below

[cc lang='sql' ]CREATE TABLE `mysql.profile_rec_extents` (
`PROF_VALUE_RECS_ID` BIGINT(20) NOT NULL AUTO_INCREMENT,
`PROFILE_RECS_ID` BIGINT(20) DEFAULT NULL,
`MAX_VALUE` VARCHAR(250) DEFAULT NULL,
`MIN_VALUE` VARCHAR(250) DEFAULT NULL,
`NUM_NULLS` BIGINT(20) DEFAULT NULL,
`MAX_LENGTH_CHARS` BIGINT(20) DEFAULT NULL,
`MIN_LENGTH_CHARS` BIGINT(20) DEFAULT NULL,
`MAX_LENGTH_BYTES` BIGINT(20) DEFAULT NULL,
`MIN_LENGTH_BYTES` BIGINT(20) DEFAULT NULL,
PRIMARY KEY (`PROF_VALUE_RECS_ID`)
) ENGINE=INNODB [/cc]

The following query returns somewhat useful information that could be used to populate the above table, but you’d have to loop it through every field.

[cc lang='sql' ]

SELECT MIN(FIELD_NAME) AS MIN_VALUE,
MAX(FIELD_NAME) AS MAX_VALUE,
MAX(CHAR_LENGTH(FIELD_NAME)) AS MAX_CHARS,
MIN(CHAR_LENGTH(FIELD_NAME)) AS MIN_CHARS,
MAX(LENGTH(FIELD_NAME)) AS MAX_BYTES,
MIN(LENGTH(FIELD_NAME)) AS MIN_BYTES
FROM profiler_recs;[/cc]

Then, you do something like: for each VARCHAR field, where MAX_LENGTH <= 25

do
[cc lang='sql' ]SELECT
COUNT(*)
, `FIELDX`
FROM
`TABLEY`
GROUP BY `FIELDX`
ORDER BY COUNT(*) DESC;[/cc]

 

and load it into something like

[cc lang='sql' ]CREATE TABLE `mysql.profile_value_recs` (
`PROF_DOMAIN_RECS_ID` BIGINT(20) NOT NULL AUTO_INCREMENT,
`PROFILE_RECS_ID` BIGINT(20) DEFAULT NULL,
`VALUE` VARCHAR(250) DEFAULT NULL,
`COUNT_VALUE` BIGINT(20) DEFAULT NULL,
`RUN_DATETIME` TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`PROF_DOMAIN_RECS_ID`)) ENGINE=INNODB [/cc]

PostGIS Install and Cloning Ubuntu Systems Notes

[cc lang='bash' ]sudo apt-get remove –purge[/cc]

[cc lang='bash' ]apt-get install deborphan debfoster
debfoster
deborphan
deborphan –guess-all[/cc]

[cc lang='bash' ]apt-get autoremove
sudo apt-get remove –purge postgresql-client
sudo apt-get remove –purge postgresql-client-8.4
sudo apt-get remove –purge postgresql-client-common
apt-get clean[/cc]

still didn’t work.
[cc lang='bash' ]jcz@dell390:/usr/local/src/postgis-1.5.2$ ls -l /usr/bin/pg*
-rwxr-xr-x 1 root root 26260 2011-02-02 03:56 /usr/bin/pg
-rwxr-xr-x 1 root root 25912 2011-04-20 10:27 /usr/bin/pg_config
-rwxr-xr-x 1 root root 13860 2010-07-06 20:21 /usr/bin/pgrep
jcz@dell390:/usr/local/src/postgis-1.5.2$ sudo rm /usr/bin/pg_config[/cc]

then
[cc lang='bash' ]sudo ln -s /usr/local/pgsql/bin/pg_config /usr/bin/pg_config
apt-get install libxml2-dev[/cc]

and finally got
[cc lang='bash' ]PostGIS is now configured for i686-pc-linux-gnu

————– Compiler Info ————-
C compiler:           gcc -g -O2
C++ compiler:         g++ -g -O2

————– Dependencies ————–
GEOS config:          /usr/local/bin/geos-config
GEOS version:         3.2.2
PostgreSQL config:    /usr/bin/pg_config
PostgreSQL version:   PostgreSQL 9.0.4
PROJ4 version:        47
Libxml2 config:       /usr/bin/xml2-config
Libxml2 version:      2.7.7
PostGIS debug level:  0

——– Documentation Generation ——–
xsltproc:
xsl style sheets:
dblatex:
convert:           [/cc]

[cc lang='bash' ]#!/bin/sh
sudo ln -s /usr/local/pgsql/bin/createlang /usr/bin/createlang
sudo ln -s /usr/local/pgsql/bin/dropdb /usr/bin/dropdb
sudo ln -s /usr/local/pgsql/bin/initdb /usr/bin/initdb
sudo ln -s /usr/local/pgsql/bin/pg_ctl /usr/bin/pg_ctl
sudo ln -s /usr/local/pgsql/bin/createdb /usr/bin/createdb
sudo ln -s /usr/local/pgsql/bin/createuser  /usr/bin/createuser
sudo ln -s /usr/local/pgsql/bin/pg_dump /usr/bin/pg_dump
sudo ln -s /usr/local/pgsql/bin/pgsql2shp /usr/bin/pgsql2shp
sudo ln -s /usr/local/pgsql/bin/pg_upgrade /usr/bin/pg_upgrade
sudo ln -s /usr/local/pgsql/bin/shp2pgsql /usr/bin/shp2pgsql

/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data >logfile 2>&1 &
/usr/local/pgsql/bin/createdb test
/usr/local/pgsql/bin/psql test

make
make install
createlang plpgsql yourtestdatabase
psql -d yourtestdatabase -f postgis/postgis.sql
psql -d yourtestdatabase -f spatial_ref_sys.sql[/cc]

how to generate a list of installed packages and use it to reinstall packages
[cc lang='bash' ]sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg –get-selections | grep -v deinstall | awk ‘{print $1}’ > 164.ubuntu-files_b.txt
sudo cat 164.ubuntu-files_b.txt | xargs sudo aptitude install[/cc]

NOTE: WordPress interprets two dashes (- -) as one dash (–). When you’re putting this into your CLI, make sure it’s dropping two dashes ‘- -’ without the space between them.

———————————————————————–

Update 20-Aug-2011

One liners install process instructions for Ubuntu 11

wget http://postgis.refractions.net/download/postgis-1.5.3.tar.gz
wget http://wwwmaster.postgresql.org/download/mirrors-ftp/source/v9.0.4/postgresql-9.0.4.tar.gz
wget http://download.osgeo.org/geos/geos-3.3.0.tar.bz2
wget http://download.osgeo.org/proj/proj-4.7.0.tar.gz

 

sudo apt-get install libreadline6-dev zlib1g-dev libxml2 libxml2-dev bison openssl libssl-dev
sudo apt-get yum install -y


mkdir -p /usr/local/src

cd /usr/local/src

tar zxvf postgresql-8.3.7.tar.gz

cd postgresql-8.3.7

./configure –with-openssl –enable-integer-datetimes

make

make install

cd /usr/local/src/postgresql-8.3.7/contrib/

make all

make install

cp /usr/local/src/postgresql-8.3.7/contrib/start-scripts/linux /etc/init.d/postgresql

chmod 775 /etc/init.d/postgresql

update-rc.d /etc/init.d/postgresql defaults

adduser postgres -d /usr/local/pgsql

echo ‘PATH=$PATH:/usr/local/pgsql/bin; export PATH’ > /etc/profile.d/postgresql.sh
echo ‘MANPATH=$MANPATH:/usr/local/pgsql/man; export MANPATH >> /etc/profile.d/pgmanual.sh

chmod 775 /etc/profile.d/postgresql.sh
chmod 775 /etc/profile.d/pgmanual.sh

mkdir -p /var/log/pgsql
chown -R postgres:postgres /var/log/pgsql/

mkdir /usr/local/pgsql/data

chown -R postgres:postgres /usr/local/pgsql/data

su – postgres

/usr/local/pgsql/bin/initdb -U postgres -E=UTF8 /usr/local/pgsql/data

These steps should be done as the postgres user.  As root, issue: `su – postgres` (no password needed), the postgresql.conf and pg_hba.conf configuration files are located in /usr/local/pgsql/data/

Using the ‘nano‘ editor, (or vi), modify the postgresql.conf to allow the installation to listen for remote connections. Also, while we’re in here let’s configure the logging to create the log file in /var/log/pgsql/. The main cause of not being able to connect to a PostgreSQL database is because of a misconfiguration in this file.

listen_addresses = ‘*’
port = 5432
log_destination = ‘stderr’
logging_collector = on
log_directory = ‘/var/log/pgsql/’
log_filename = ‘postgresql-%Y-%m-%d’
log_line_prefix = ‘ %t %d %u ‘

Now, edit the pg_hba.conf and configure some network rules. Add the line in red to match your LAN address range. Set access from other computers to use md5 authentication. You can also set the other methods to md5, (and others) but for managability, leave the local connections set to ‘trust’ for now. The order of rules in this file matters.

# TYPE DATABASE USER CIDR-ADDRESS METHOD
# “local” is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
host all all 192.168.0.0/24 md5
# IPv6 local connections:
host all all ::1/128 trust

/etc/init.d/postgresql start

 

My MySQL latest config file

I never record this stuff and I always wish I did. So here’s a working MySQL config file that I’m using on a linux virtual machine with 2GB of memory. Notes and all so I don’t keep having to look stuff up.

# MySQL config file for APPSRV VPS with 2GB of memory
# 25-March 25-2011
# jcz.
# For MySQL 5.1
# The following options will be passed to all MySQL clients
[client]
port        = 3306
socket        = /var/lib/mysql/mysql.sock

Continue reading

Benchmarking MySQL

So I’m trying to figure which old machine to turn into a little mySQL number cruncher. So, I’m going to do some clean installs of Ubuntu server on each and run this little script (with the same my.cnf) and see how they fair. Perhaps you will this useful, run it a few times.

[cce_bash]

#!/bin/sh
# jcz 2011-April-22
#
# This script will time your MySQL database in a repeatable way
#
# Date and other variables pretty self explanatory, S is seconds
# date format is currently YYYYMMDD_HHMMSS
    dater=$(date +%Y%m%d_%H%M%S)
    dayer=$(date +%a)
    myhost=$(hostname)
    directory=$(pwd)
    outfile=”slapout.txt”

# THE MYSQL USER
    super=”username”

# THE MYSQL SUPERPWORD
    superword=”password”

# THE MYSQL HOSTNAME or IP.
    hoster=”localhost”

echo “—————— BEGIN ——————–” >> $outfile
date >> $outfile
echo $myhost >> $outfile
echo $directory >> $outfile

# COPY THE COMMAND BELOW
mysqlslap -u$super -p$superword  -h$hoster -v –concurrency=1 –iterations=2 –number-int-cols=4 –number-char-cols=5 –auto-generate-sql –auto-generate-sql-secondary-indexes=3 –engine=myisam,innodb  –auto-generate-sql-add-autoincrement –auto-generate-sql-load-type=mixed  –number-of-queries=2 >> $outfile

 echo “…………………………..” >> $outfile
 # PASTE THE COMMAND BELOW BETWEEN THE QUOTES OR EDIT BOTH. I CAN’T FIND ANOTHER WAY TO RECORD IT
echo “mysqlslap -u$super -p$superword  -h$hoster -v –concurrency=1 –iterations=2 –number-int-cols=4 –number-char-cols=5 –auto-generate-sql –auto-generate-sql-secondary-indexes=3 –engine=myisam,innodb  –auto-generate-sql-add-autoincrement –auto-generate-sql-load-type=mixed  –number-of-queries=2″ >> $outfile
echo “…………………………..” >> $outfile
echo “The above command was executed to produce the results above it.” >> $outfile
echo “—————— END ——————–” >> $outfile
echo “” >> $outfile
echo “” >> $outfile
echo “” >> $outfile

[/cce_bash]

The above script makes output like below

Continue reading

Tables of Tables from MySQL

From Matthew Crowley on the MySQl forums (http://forums.mysql.com/read.php?101,8004), this php script will output a DESC of all databases and tables in MySQL. It needs some formatting for the output, but it works and might be handy later. I really just need to get around to figuring out how to do this in a procedure or something.

[cce_php]

$connection = mysql_connect(“localhost”,”root”,”PASSWORD”);

$showDB = mysql_query(“show databases”);
if($myrow=mysql_fetch_array($showDB))
{
do
{
$DB = $myrow["Database"];
echo “$DB \n”;
$showTable = mysql_query(“show tables from $DB”);
if($myrow=mysql_fetch_array($showTable))
{
do
{
$col = “Tables_in_”.$DB;
$Table = $myrow["$col"];
echo “$Table \n”;
$describeTable = mysql_query(“describe $DB.$Table”);
if($myrow=mysql_fetch_array($describeTable))
{
do
{
$field = $myrow["Field"];
$type = $myrow["Type"];
$null = $myrow["Null"];
$key = $myrow["Key"];
$default = $myrow["Default"];
$extra = $myrow["Extra"];
echo “$field \t $type \t $null \t $key \t $default \t $extra \n”;
}
while ($myrow=mysql_fetch_array($describeTable));
}
}
while ($myrow=mysql_fetch_array($showTable));
}
}
while ($myrow=mysql_fetch_array($showDB));
}

[/cce_php]

Handy database documenter for MySQL

UPDATE: see the next iteration on this project [here]

 

Here’s a view that will spit out just about everything MySQL (5.1) knows about the tables and fields it maintains for you. The first field can be joined to the output of something like

[cce_sql]

SELECT * FROM AZ_CA_NV_UT_species_LOCAL PROCEDURE ANALYSE(10000, 4000);

[/cce_sql]

to see before and after “optimal” (1)   (2) field types and lengths predicted by the internal PROCEDURE ANALYSE.

[cce_sql]

CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v_field_table_data` AS
SELECT
CONCAT_WS(‘.’,`tables`.`TABLE_SCHEMA`,`columns`.`TABLE_NAME`,`columns`.`COLUMN_NAME`) AS `FIELD_NAME`,
`tables`.`TABLE_COMMENT`             AS `TABLE_COMMENT`,
`columns`.`TABLE_NAME`               AS `TABLE_NAME`,
`columns`.`COLUMN_NAME`              AS `COLUMN_NAME`,
`columns`.`COLUMN_TYPE`              AS `COLUMN_TYPE`,
`columns`.`DATA_TYPE`                AS `DATA_TYPE`,
`columns`.`COLUMN_KEY`               AS `COLUMN_KEY`,
`tables`.`TABLE_SCHEMA`              AS `TABLE_SCHEMA`,
`tables`.`TABLE_TYPE`                AS `TABLE_TYPE`,
`tables`.`ENGINE`                    AS `ENGINE`,
`tables`.`VERSION`                   AS `VERSION`,
`tables`.`ROW_FORMAT`                AS `ROW_FORMAT`,
`tables`.`TABLE_ROWS`                AS `TABLE_ROWS`,
`tables`.`AVG_ROW_LENGTH`            AS `AVG_ROW_LENGTH`,
`tables`.`DATA_LENGTH`               AS `DATA_LENGTH`,
`tables`.`MAX_DATA_LENGTH`           AS `MAX_DATA_LENGTH`,
`tables`.`INDEX_LENGTH`              AS `INDEX_LENGTH`,
`tables`.`AUTO_INCREMENT`            AS `AUTO_INCREMENT`,
`tables`.`CREATE_TIME`               AS `CREATE_TIME`,
`tables`.`UPDATE_TIME`               AS `UPDATE_TIME`,
`tables`.`TABLE_COLLATION`           AS `TABLE_COLLATION`,
`tables`.`CHECKSUM`                  AS `CHECKSUM`,
`tables`.`CREATE_OPTIONS`            AS `CREATE_OPTIONS`,
`columns`.`ORDINAL_POSITION`         AS `ORDINAL_POSITION`,
`columns`.`COLUMN_DEFAULT`           AS `COLUMN_DEFAULT`,
`columns`.`IS_NULLABLE`              AS `IS_NULLABLE`,
`columns`.`CHARACTER_MAXIMUM_LENGTH` AS `CHARACTER_MAXIMUM_LENGTH`,
`columns`.`CHARACTER_OCTET_LENGTH`   AS `CHARACTER_OCTET_LENGTH`,
`columns`.`NUMERIC_PRECISION`        AS `NUMERIC_PRECISION`,
`columns`.`NUMERIC_SCALE`            AS `NUMERIC_SCALE`,
`columns`.`CHARACTER_SET_NAME`       AS `CHARACTER_SET_NAME`,
`columns`.`COLLATION_NAME`           AS `COLLATION_NAME`,
`columns`.`EXTRA`                    AS `EXTRA`,
`columns`.`PRIVILEGES`               AS `PRIVILEGES`,
`columns`.`COLUMN_COMMENT`           AS `COLUMN_COMMENT`,
NOW()                                AS `RUN_DATETIME`
FROM (`information_schema`.`tables`
JOIN `information_schema`.`columns`
ON (((`tables`.`TABLE_SCHEMA` = `columns`.`TABLE_SCHEMA`)
AND (`tables`.`TABLE_NAME` = `columns`.`TABLE_NAME`))))

[/cce_sql]

MySQL Functions

I just want to record these for future reference. I’m actually using the first now.

From the MySQL 5.0 Online manual

An example of how to make the first letter in a string uppercase – analogous to UCFIRST

[cce_sql]

SELECT CONCAT(UPPER(SUBSTRING(firstName, 1, 1)), LOWER(SUBSTRING(firstName FROM 2))) AS properFirstName

[/cce_sql]

a user-defined function in MySQL 5.0+ similar to PHP’s substr_count(), since I could not find an equivalent native function in MySQL. (If there is one please tell me!!!)

[cce_sql]

delimiter ||
DROP FUNCTION IF EXISTS substrCount||
CREATE FUNCTION substrCount(s VARCHAR(255), ss VARCHAR(255)) RETURNS TINYINT(3) UNSIGNED LANGUAGE SQL NOT DETERMINISTIC READS SQL DATA
BEGIN
DECLARE count TINYINT(3) UNSIGNED;
DECLARE offset TINYINT(3) UNSIGNED;
DECLARE CONTINUE HANDLER FOR SQLSTATE ’02000′ SET s = NULL;

SET count = 0;
SET offset = 1;

REPEAT
IF NOT ISNULL(s) AND offset > 0 THEN
SET offset = LOCATE(ss, s, offset);
IF offset > 0 THEN
SET count = count + 1;
SET offset = offset + 1;
END IF;
END IF;
UNTIL ISNULL(s) OR offset = 0 END REPEAT;

RETURN count;
END;

||

delimiter ;

[/cce_sql]

Use like this:

[cce_sql]

SELECT substrCount(‘/this/is/a/path’, ‘/’) `count`;

[/cce_sql]

`count` would return 4 in this case. Can be used in such cases where you might want to find the “depth” of a path, or for many other uses.

It’s pretty easy to create your own string functions for many examples listed here

[cce_sql]
## Count substrings

CREATE FUNCTION substrCount(x varchar(255), delim varchar(12)) returns int
return (length(x)-length(REPLACE(x, delim, ”)))/length(delim);

SELECT substrCount(‘/this/is/a/path’, ‘/’) as count;

[/cce_sql]

+——-+
| count |
+——-+
|     4 |
+——-+

[cce_sql]

SELECT substrCount(‘/this/is/a/path’, ‘is’) as count;

[/cce_sql]

+——-+
| count |
+——-+
|     2 |
+——-+

[cce_sql]

## Split delimited strings

CREATE FUNCTION strSplit(x varchar(255), delim varchar(12), pos int) returns varchar(255)
return replace(substring(substring_index(x, delim, pos), length(substring_index(x, delim, pos – 1)) + 1), delim, ”);

select strSplit(“aaa,b,cc,d”, ‘,’, 2) as second;

[/cce_sql]

+——–+
| second |
+——–+
| b      |
+——–+

[cce_sql]

select strSplit(“a|bb|ccc|dd”, ‘|’, 3) as third;

[/cce_sql]

+——-+
| third |
+——-+
| ccc   |
+——-+

[cce_sql]

select strSplit(“aaa,b,cc,d”, ‘,’, 7) as 7th;

[/cce_sql]

+——+
| 7th  |
+——+
| NULL |
+——+

[cce_sql]

## Upper case first letter, UCFIRST or INITCAP

CREATE FUNCTION ucfirst(x varchar(255)) returns varchar(255)
return concat( upper(substring(x,1,1)),lower(substring(x,2)) );

select ucfirst(“TEST”);

[/cce_sql]

+—————–+
| ucfirst(“TEST”) |
+—————–+
| Test            |
+—————–+

[cce_sql]

##Or a more complicated example, this will repeat an insert after every nth position.

drop function insert2;
DELIMITER //
CREATE FUNCTION insert2(str text, pos int, delimit varchar(124))
RETURNS text
DETERMINISTIC
BEGIN
DECLARE i INT DEFAULT 1;
DECLARE str_len INT;
DECLARE out_str text default ”;
SET str_len=length(str);
WHILE(i<str_len) DO
SET out_str=CONCAT(out_str, SUBSTR(str, i,pos), delimit);
SET i=i+pos;
END WHILE;
– trim delimiter from end of string
SET out_str=TRIM(trailing delimit from out_str);
RETURN(out_str);
END//
DELIMITER ;

select insert2(“ATGCATACAGTTATTTGA”, 3, ” “) as seq2;

[/cce_sql]

+————————-+
| seq2                    |
+————————-+
| ATG CAT ACA GTT ATT TGA |
+————————-+

—————————-

Accessing versioned geodata in ArcSDE with SQL

This is excerpted from the following forum post:

“We’re running SDE and have several featuresets with versioning turned on…. We have numerous non-GIS applications that use SQL queries to access information from the spatial data. However, we don’t get all the features that I’m expecting to be returned. How do I access the features that have been added/deleted (and are, therefore, “hidden” from a straight SQL query)?”

As long as it is just the attributes that you are after, you can set up a multi-versioned view using sdetable.exe.

sdetable.exe -o create_mv_view -T mvv_wMeter -t wMeter

Then on your db connection, you execute the set_current_version stored proc to set the version, then issue your select statement on the multi-versioned view:

exec sde.set_current_version 'SDE.SOMEOTHERVERSION'
GO
Select COUNT(*) from mvv_wMeter ;

i don’t have it my example, but sdetable.exe needs db connection info arguments: -s,-i,-u,-p,-D in some combination depending on your config (I usually just set the SDEDATABASE,SDEINSTANCE,SDESERVER env vars in a batch file–sdetable use the env vars if they exist). – Jay Cummins Aug 6 ’10 at 19:39

I think this ESRI support page may be relevant.

I believe it is worth posting the warnings on that page here:

Never use database management system (DBMS) tools to update any row ID (object ID) field maintained by ArcSDE in the database. These object ID fields are allocated and managed by the geodatabase and, therefore, should not be altered using SQL.

Never edit the DEFAULT version of the geodatabase using SQL. Starting an edit session on a version obtains an exclusive lock on the state that the version references. If you lock the DEFAULT version, you prevent ArcGIS users from connecting to the geodatabase.

In the 9.3 help page they also warned against editing non-simple feature class attributes (Geometric Networks, Topologies, etc.) via SQL.

Remove header from a text file

I’m processing some Darwin-schema biodiversity text files in preparation for GISing them and I need to trim the headers from the tops of the files before importing them to the database. Of course, *NIX command line utilities have been my friend here. The following will trim the first line off the file. Just up the number to trim more lines off. I’m using Cygwin on Windows to run these.

sed '1d' file.txt # trims the line then replaces file.txt, so be careful.

awk 'FNR&gt;1{print}' file.txt &gt; newfile.txt # I ended up using this to be safe. It was easier to test.

tail -n+2 /path/to/file &gt; /path/to/output # can be tested with echo -e "foo\nbar\nbaz" | tail -n+2

Also in my case, I wanted column headers left in file, and thankfully the headers were pretty much the same across all the files. So I grep searched in the directory with all the files for the name of the first uniquely worded column and asked grep to also return the line numbers of where it found that word. Then I use that number (minus one) to put into my header deleting command.

&gt; grep -n 'Institution' *
HerpnetUtahBottomHalfProviders_January25_2010.txt:35:Institution	Collection	Catalog number text ...
HerpnetUtahTopHalfProviders_January25_2010.txt:27:Institution	Collection	Catalog number text...

Then, for example, I would use

 awk 'FNR&gt;34{print}' HerpnetUtahBottomHalfProviders_January25_2010.txt &gt; HerpnetUtahBottomHalfProviders_January25_2010_trimmed.txt

I love using Excel to craft a slew of commands then copy/paste them into the shell to batch process tens of files in my case.