I'm developing a database that I'm going to routinely import CSV data into. The CSV data doesn't contain a primary key and I have no control over the source of the CSV so I'm stuck with it as it is.
The data has a first name column, last name column and date column which combined will probably be unique 99.9% of the time. There are then 10 other columns which may be updated over time.
So my issues are I need to be able to import new records, not import duplicates into my database and detect changes in existing records.
I could use the first name, last name and date columns as a composite key but I'm thinking maybe I could generate a hash based on the combined data of those columns and then create a hash based on the combined data of the other columns. I'd then be able to recognise existing records and then check for changes to those records using the hashes rather than have to check column by column or update the whole lot 'just in case'.
I'm pretty sure this would work - does anyone have any comments, suggestions or better ideas?
The data has a first name column, last name column and date column which combined will probably be unique 99.9% of the time. There are then 10 other columns which may be updated over time.
So my issues are I need to be able to import new records, not import duplicates into my database and detect changes in existing records.
I could use the first name, last name and date columns as a composite key but I'm thinking maybe I could generate a hash based on the combined data of those columns and then create a hash based on the combined data of the other columns. I'd then be able to recognise existing records and then check for changes to those records using the hashes rather than have to check column by column or update the whole lot 'just in case'.
I'm pretty sure this would work - does anyone have any comments, suggestions or better ideas?