Quantcast
Channel: dBforums – Everything on Databases, Design, Developers and Administrators
Viewing all 13329 articles
Browse latest View live

Database design - If yes, then what date...

$
0
0
I am not a database designer or programmer, but I am modeling, as best as I can an new database using MySQL Workbench so that I can pass it on to the database developer...

Here's my question...I am creating my tables, and have a field A that is a yes/no answer...The next field B is a date field..."If row A is "yes", then provide the date in field "B"...

Since field B is dependent on the answer to Field A, is this just issue addressed in the code, or do I need do something special with the fields in the table? So if Field A is yes, then a date is required in Field B, otherwise field B is blank...

I know this is a trivial question, so I appreciate your patience and your help...

Thanks.

translate function

$
0
0
Hello,
Can you please let me know what exactly this function is doing here. we are exporting from DB2 and later will import to teradata. This function is being used in the export query to export the column1.

TRANSLATE(column1,'',x'000A0D')

thank you

Changing specific value wthin a feild

$
0
0
Hello BD users,

I have another simple question that I am trying to solve. I would like to change the word Street to ST within the address line field for all address lines for example from 123 Main Street to 123 Main ST. However, my script change entire field to ST. Is there away to ONLY change Street to ST but not the entire text?

Thanks!
Mohammed

Not in Kansas anymore. Here's my setup.. Experience? Suggestions?

$
0
0
** note To start out, let me just say that when it comes to computers I'm completely self taught. I stated learning web development and database about 2 months ago.g
I posted this yesterday, but it didn't show up.

Its longer than I intended it to be, if you don't want to read the whole thing skip to the table diagrams and "My Concerns so far".

What I'm Doing
Buidling an advanced online metering system which for the most part is easier than i thought (except for the database stuff, which i think i'll get eventually)

How I'm Doing it now
The device is currently uploading 123 columns of floats* and a datetime (time stamp). This is uploaded/inserted every 2 seconds for a total of 43,200 rows per day.
The device is also uploading calculated aggregations(sums, min, max, average, and some trig and integrals) for time periods of 60s (1 min) 300s and (5 min) 900s all of which are calculated off of the 2 second data.
additional tables with event data, 1 row per event, and daily statistics(single row per day)
*(values range from 0 to 1000000 rounded to 5th decimal place .00001)

example database layout per meter:
Code:

# LAYOUT PER DEVICE
meter_data          <- database name
----meter1_2ss      <- 2 second data table with 124 columns
----meter1_60ss      <- 60 second data table with the same 124 columns plus 86 more
----meter1_300ss    <- 300 second data table with the same 124 columns plus 86 more
----meter1_900ss    <- 900 second data table with the same 124 columns plus 86 more
----meter1_events    <- events table with haven't designed yet
----meter1_daily    <- daily statistics that can't be easily or quickly calculated off of the data

---- **currently there are a set of tables like this for each device**

Extra information
I'm using Innodb with barracuda and file_per_table=on
with this setup 1 months worth of data(for all tables for one device) comes in at just over 1.3 million rows and a file size (with barracuda compression enabled) of about 200 mb. By rebasing and other methods i can probably shrink this to 100mb
For reference I leaned everything I know about databases on this project in the last 2 months.
Goal is to keep data in the database for as long as possible, and when necessary move it to a compressed file structure(or store a days worth of data in a single row/column of a different database).
Data compression per row of data(in csv format) yields very small gains; 374 bytes to 299 b or 20%; but per 100 rows(again in csv)it goes from 35.9kb to 3.9kb (about 90%, at 800,000 rows it maxes out at 93%).
That was with lzma compression, zip is about 4-6% less on average.
Try and store with database and archive at least 2 years of data per meter

Tools i'm using
MySQL 5.x not sure which, don't know enough to know what to use.
Python SQL alchemy (just for table creation for now, it seams ORM's are just as hard or harder to learn than SQL, especially when you don't know much about db's or SQL)
official python MySQL connector with compressed protocol and prepared statements

My Concerns so far

  1. For now with 2-10 devices this works, well what about when I have more, 100's 1000's
  2. At 100 devices that will have the database processing 4.43 million inserts a day (120GB database size increase per year) Can a database handle that many inserts? If not is there a technology that would?
  3. Is MySQL the right database for this? Would something like Postgres or Mongo be better?
  4. should i be using mysql database connector to upload rows to the database, (will that be stopped by some firewalls and such) or is there another method
  5. is this scalable to 10000+ devices (45 million+ inserts per day, 1.2TB per year
  6. Would it be more cost effective at 1000+ devices to host my own database on my own hardware?
  7. I've come across schema designs where all the time-series data was placed in a single table with device identifiers. Would than work for my database? is it advisable? would it perform okay? (http://stackoverflow.com/questions/4...ational-or-non) That link suggests that storing all data in a single table as illustrated below would be a good Idea.

Setup 1
Code:

meter_data        <- database name
----clients        < columns    (customer, location, device_ids) device_ids will probably be some kind of list or dictionary as there an be more than 1 device per location
----mdata_2        <- 124 colS + (device_id,...) 2 second data only
----mdata_rest    <- 210 colS + (device_id, data_freq,) this would hold the 60, 300, and 900 second data for all devices
----mdata_events  <- columns    (device_id, ...)
----mdata_daily    <- columns    (device_id, ...)

Setup 2
the same with mdata_2 and mdata_rest combined in a single table.
both tables have 124 columns that are the same, the mdata_rest table just has an additional 86 columns.
I could merge these into a single table and just null (or 0?) out the columns that mdata_2 doesn't have. Illustrated below.

Code:

# 2 second data:
 |datetime, device_id, data1, data2, data3, ... data122, data123|
 | 2015:.., 000000001, val 1, val 2, val 3, ... val 122, val 123|
| ^index^|  ^index^ |

# 60, 300, 900 second data:
 |datetime, device_id, data_freq, data1, data2, data3, ... data122, data123, data124, ... data209, data210|
 | 2015:.., 000000001,      300, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
 | 2015:.., 000000001,      900, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
 | 2015:.., 000000001,        60, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
| ^index^|  ^index^ |  ^index^ |

# NEW TABLE WITH BOTH MERGED INTO 1
 |datetime, device_id, data_freq, data1, data2, data3, ... data122, data123, data124, ... data209, data210|
 | 2015:.., 000000001,      300, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
 | 2015:.., 000000001,      900, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
 | 2015:.., 000000001,        60, val 1, val 2, val 3, ... val 122, val 123, val 124, ... val 209, val 210|
 | 2015:.., 000000001,        2, val 1, val 2, val 3, ... val 122, val 123,    null, ...    null,  null |
 | ^index^|  ^index^ |  ^index^ |                                from this ^ point on its all nulls


Thanks
Jake

Concat is not workgin in inclause condition.

$
0
0
Hi All,

If i give values in inclause directly results are coming fine(2 rows) as like below, but if i use concat instead of values, its returning only one row and that is not the expected output, please help me in this regard.

mysql> select c_id,c_name from category where c_id in (870,854);
+------+---------------+
| c_id | c_name |
+------+---------------+
| 854 | Telugu |
| 870 | Telugu Events |
+------+---------------+
2 row in set (0.04 sec)

mysql> select c_id,c_name from category where c_id in (select concat(870,',',854) as c_id);

+------+---------------+
| c_id | c_name |
+------+---------------+
| 870 | Telugu Events |
+------+---------------+
1 row in set (0.04 sec)



Thanks.

New "Brilliant Database Online" cloud service will be presented on May 12

$
0
0
Any News or did I miss something>

Export file using Sybase Event

$
0
0
I'm trying to output a file to a shared folder, I can make the output but I can't run it on a event/Stored Procedure.

The code snippet that I have to the output is this:

Code:

BEGIN
    select * from table1; output to 'c:\\teste.txt' format text with COLUMN names
END

But when I include this in a event, the sybase gives me this error.

[Sybase][ODBC Driver][SQL Anywhere]Syntax error near 'output' on line 2 SQLCODE: -131 SQLSTATE: 42000 SQL Statement: ALTER EVENT "dba"."Export" HANDLER BEGIN select * from table1; output to 'c:\teste.txt' format text with format text with COLUMN names END

I am using Sybase Central with sql anywhere 12

Tables, Combo-box and auto population based on a different combo selection on form

$
0
0
Hi everyone,
What is the best practice of storing lists of Subdivisions/Complexes for each town of a state. I would like on the Prospect entry form for the subdivisions to be populated in a combo based only on the selected township in a different combo on that form.

Example -
User selects Township and the Subdivision Combo should populate with only subdivisions in that particular town.


Notes
Township combo on form is sourced from a query displaying for example - Howell, NJ 07731.
Not every Street is in a subdivision so I also need to have a None option for each subdivision in each town.


Would I create a table with a column for each town with subdivisions listed under that town then use code to have the Subdivision combo populate by column or is there a better way? I would definitely need help with the VBA but I know there is a different section of the forum for VBA. At this point it is purely design.

I am currently using a combo on the Prospecting form and adding new subdivisions as needed but this is very impractical, the list is getting long and I can't sort the subdivisions.

Thanks for all your help in advance.

Sybase ODBC returning datetime as string, but saying it's a timestamp

$
0
0
I'm trying to query against Sybase ASE 15 using ODBC. A couple of the columns in the table have user-defined types that ultimately map to datetime, though the types have different definitions. When queried, ODBC indicates that both are SQL_TYPE_TIMESTAMP, so my program binds both as SQL_C_TYPE_TIMESTAMP. This works for one of the columns, but Sybase fills the other one in with a stringified date, instead. Needless to say, the result is a timestamp struct filled with gibberish.

The two user types, plus a dependent definition:

Type_name Storage_type Length Prec Scale Nulls Default_name Rule_name Access_Rule_name Identity
VDT_DATETIME datetime 8 NULL NULL 1 <null> <null> <null> 0

Type_name Storage_type Length Prec Scale Nulls Default_name Rule_name Access_Rule_name Identity
VDT_DATETIMESTAMP datetime 8 NULL NULL 0 VDT_DATETIME_DEF <null> <null> 0

Name Owner Object_type Create_date
VDT_DATETIME_DEF dbo default Apr 29 2003 12:37PM

The VDT_DATETIMESTAMP is the type that works. The other one, VDT_DATETIME, only works if you explicitly CONVERT it to datetime. Otherwise, as I said, it returns a string, but lies and says that it's a timestamp.

I looked for some way that my program could know to bind the two differently, not wanting to add some hack (which I could probably get to work).

I'm not sure why have Default_name set makes one of these work, and not the other. Can anyone shed light on this?

Thanks,

James.

SQL database error 823

$
0
0
Hi everyone!
I have a question….How to fix sql database when generating an error 823?
Please help me to out this!

SQL to retrieve the latest record

$
0
0
I'm trying to write a query to achieve the following result for the sample data below. Any help is much appreciated.

Table-A

ID FK Dt
DD1 D4Q null

Table-B

Id Num Cd
D4Q 0 AA1
D4Q 1 AA2
D4Q 2 AA3

Expected Result Set /* Record with Max value in Num need to be retrieved.

ID Fk Num Cd
DD1 D4Q 2 AA3

Whitelisting IIS

$
0
0
Does anyone know how to create an effective "whitelist" in IIS??

I know I can do distinct and specific IPs
And I know I can restrict a set of ambiguous IPs and dynamic IPs in an open portal. (subnets, etc)

But I'm unaware if you can open for a set or dynamic IPs within an otherwise restricted/closed portal.

Any insights from our other IT gurus out there??

Questions on Update statement on a table in DPF.

$
0
0
I have a table that is created in a DPF environment, this table is spread across multiple partitions hashed based on the PK
(column A).

I have an update statement that runs on this table, the update is done using an ETL tool (Datastage V8.5). On an average daily
input record volume for update is 50 million records. The update statement is happening based on the columns that are non-indexed and
non-PK column (Columns B,C,D and E). The datastage job to perform this update is running very slow and the bottle neck seems to be in the DB update.
Now My questions are

1. There is an option to run the DB2 connector stage in parallel and also the ETL nodes can communicate directly with
the DB2 nodes where the data is present. If in case, a wrong partitioning was done in ETL and data for update ends up in a
different Db2 node than where the actual data is, there wont be updates happening properly right?

2. The DB is in a different server than the actual Datastage server. Both are running AIX unix.
Now if login to DB2 from DS application server using Command line and execute a query on partitioned table, how does that execute?
Will it also be executed in parallel and run on all the partitions that the table is in? (if there are more than one table referenced
and the tables are in different partitions, there would be data movement within partitions or they are moved to some common space
for processing?)

3. Suppose if I am joining two tables that are stored in different database partitions based on column C and say that
table A is partitioned using Hashing of column A and table B using hashing of column B. How will the join happen across
data stored in different database partitions? Would it come to some spool space for joining across partitions?
Will this result in performance degradation?
(if you could point me any books related to this it would be helpful)
Since I have worked on Teradata earlier, I am just trying to compare Teradata with DB2 in this aspect.

4. How do I find out what is the Db2 co-ordinator node my datastage is connecting to? (basically in a DPF environment, all queries
executed through any application would go to co-ordinator node and then distributed to the paritions right? in the case of
data retrieval queries, the result set from different paritions would be collated again in co-ordinator node?)

5. There is a property in Datastage which can help the different nodes in datastage to connect directly with the DB2 nodes where
the data is in instead of communicating through Co-ordinator node. So if I set this property, how do I what are the db2 nodes where
this query is executed? Is there any system table where this is stored?

6. The table is currently empty but updated with 50 million rows. But still running for more than 4 hours in production.
Do the stats need to be collected on empty table as well. (The table is just created new not truncated).
What else could be the reason the update query is running slowly? What can I do to fix it ? What steps should I follow to find out the root cause of this performance issue?

7. How do I check the explain plan or timerons or performance of this query in this case? If it was table to table update
I could have taken explain plan. But here, the data is read from the file and loaded to table, so how can I take explain plan or timerons or
measure the performance while the query is running? What could be the reason for slow performance in this case?
And please let me know what do I go about analysing slow performance of a query executed from datastage?

8. Would creating an index on the columns used in where clause of update would help? I am just wondering why the query would run so slowly
on a table that has been just created. Should I do any re-orgs or runstats on this table?

sybase license not getting applied

$
0
0
Hi All,

We have completed an installation of Sybase ASE 15.7.0 around 8 month back and applied the permanent license and working fine and due to some I/O error the unix team had rebooted the OS and then we restarted the Sybase ASE as well and after that we found that license is about to expire and I have tried to copy the same backup license which is available locally with us and tried to restart the server but following are the issues arised now ...


esesbshd2 CCSOC_TEST_SDS> sysam start
Starting the SYSAM License Server using licenses from:
/opt/sybase/ase/15.7.0/SYSAM-2_0/licenses
sysam: /opt/sybase/ase/15.7.0/SYSAM-2_0/bin/lmgrd: not found

Failed to determine license server status.

Additional errors may be reported in the logfile at:
/opt/sybase/ase/15.7.0/SYSAM-2_0/log/SYBASE.log
For troubleshooting and further information, refer to the SySAM FAQ at:
http://www.sybase.com/sysam



esesbshd2 CCSOC_TEST_SDS> sysam reread
lmutil - Copyright (c) 1989-2007 Macrovision Europe Ltd. and/or Macrovision Corporation. All Rights Reserved.
lmreread failed: Cannot connect to license server system. (-15,570:146 "Connection refused")




Could any one help me in fixing this error.

Thanks
Ram.

DB2 Scheme Structure restriction

$
0
0
Hi,

I have 10 schemas and I want user "one" to access only schema no 3. My question is that I don't want that user to view the structure of any other schemas except the no 3. Is it possible?
Like on DB solo tool or any other db2 query tool, I can view all the schemas and table structure on browser directory but I want the user to view only schema 3 structure. Can it be done on the same database and not through another database/dblink.

Thanks!!

The conversion of a varchar data type to a datetime data type resulted in an out-of-

$
0
0
Hi all, hope in your help.
I have problem to execute query with interval date.

If try this query I don't have problem and the output is correct:

Code:

SELECT * FROM dotable
WHERE
        dotableDate BETWEEN CONVERT (datetime, '01/01/2015', 121)
AND CONVERT (datetime, '09/01/2015', 121);

Instead if try this I have error:

Code:

SELECT * FROM dotable
WHERE
        dotableDate BETWEEN CONVERT (datetime, '25/01/2015', 121)
AND CONVERT (datetime, '28/01/2015', 121);


[Err] 22007 - [SQL Server]The conversion of a varchar data type to a datetime data type resulted in an out-of-range value.

Why?
Can you help me?

Thank you in advance.

Officially Working w/ Databases

$
0
0
Today marks my 1st day as a 'Database Associate' in which I will be working with / under our sole MS SQL DBA. I have 90 days from today to meet the following:

- Study / take / pass the Microsoft MTA 98-364 Database Fundamentals certification (which work will pay for).

I sat and suffered through the Microsoft Virtual Training Academy which directly covers the 98-364 exam but it's like watching a really bad soap opera skit. Regardless I pulled the info from it and tried to focus on the data and not the bad video / marketing.

I have very low RDBMS skill sets and really love databases. I'm a tenured (10yr) Linux Sys Admin w/ passion for PostgreSQL, MongoDB, Oracle, and MS SQL. I'm not good w/ SQL and would say I can do basic DDL statements to create my database / schema and very basic DML statement knowledge. I'd say my DML knowledge drops around advanced SQL queries like JOINS, Stored Procedures, Indexed Views, etc etc etc.

So that's me in a nutshell. How worried should I be about getting this certification? I know that eventually over time I will cut my teeth and get a nice DBA tool belt of skills but for now I'm being given a very very generous opportunity to be a passionate DBA. Just looking for reassurance this entry level certification is easy enough w/ database and SQL skills and anything else you suggest I can read / study up on.

Reorg all tables for each active database

$
0
0
Hello Guys,

I did a shell script to generate a dynamic reorg file for all tables but I'm facing problems. The script finished with success but it's executing for all databases inside the file. When I see each logfile for example:
1- db2wfti2_reorg_DATABASE1.log(In this file the reorg try to execute in a table who only exists in database 2...so the reorg don't execute "SQL2211N The specified table does not exist."
2- db2wfti2_reorg_DATABASE1.log
3- db2wfti2_reorg_DATABASE1.log


Take a look in my script:

DB2EXEC=/db2wfti2/home/inst/sqllib/bin/db2
$DB2EXEC list active databases | grep "Database name" | awk '{print $4}' | while read DATABASE
do
REORGFILE=/db2wfti2/home/inst/scripts/REORG_$DATABASE"_"`date +"%Y%d%m%H%M%S"`.out
$DB2EXEC -x "select 'reorg table',substr(rtrim(tabschema)||'.'||rtrim(tabname ),1,50),';'from syscat.tables where type = 'T' " > $REORGFILE
LOGREORGFILE=/tmp/log/db2wfti2_reorg_$DATABASE.log
echo $DATABASE"_"`date +"%Y%d%m%H%M%S"` >> $LOGREORGFILE
$DB2EXEC connect to $DATABASE
$DB2EXEC -tvf $REORGFILE >> $LOGREORGFILE
done

Could you help me how generate all reorg table for each active database without "error" as "SQL2211N The specified table does not exist."

MON_GET_PKG_CACHE_STMT error

$
0
0
Hi,

I'm trying to get a list of SQL statements in the package cache using the follow statement but I'm getting the error below. This works on all of my internal databases and all of my clients except for one. I'm running db2 9.7 FP 3 on AIX. Does anyone have any ideas why this database is giving this error?

From config file
db2 get db cfg for mydb |grep MON|grep METRICS

Request metrics (MON_REQ_METRICS) = BASE
Activity metrics (MON_ACT_METRICS) = BASE
Object metrics (MON_OBJ_METRICS) = BASE

Statement
SELECT MEMBER,
SECTION_TYPE , TOTAL_CPU_TIME , NUM_EXEC_WITH_METRICS,
TOTAL_CPU_TIME/NUM_EXEC_WITH_METRICS as
AVG_CPU_TIME,EXECUTABLE_ID
FROM TABLE(MON_GET_PKG_CACHE_STMT ( NULL, NULL, NULL, -2)) as T
WHERE T.NUM_EXEC_WITH_METRICS <> 0
ORDER BY AVG_CPU_TIME desc
FETCH FIRST 20 ROWS ONLY;

Error
Routine "*CHE_STMT" (specific name "MON_GET_PKG_CACHE_STMT") is implemented with code in library or path "...trusted", function "monGetPkgCacheStmtv97fp6" which cannot be accessed. Reason code: "6".. SQLCODE=-444, SQLSTATE=42724, DRIVER=4.17.30

Thank you

Invalid identifier error in package but the column is valid

$
0
0
Hello All,

When I am executing Below select query, it is being executed properly & giving me desired output.

SELECT sslock.userid , sslock.plantstructureentryid , sslock.audsid
FROM sslock
WHERE sslock.audsid NOT IN ( SELECT audsid FROM v$session );

But whenever I am using same select query as a cursor in a package, then while compiling it is giving "ORA-00904: "audsid": invalid identifier".

Below I am giving the exact package. Both package & the table are from the same schema. Problem query is highlighted with bold & italic manner.

the query is running fine when we run it independently. But the moment I use it in the below package it is giving Invalid identifier error at the place of Delete statement.



create or replace
PACKAGE BODY pkg_sslock
AS

PROCEDURE removeIllegalLocks
IS

CURSOR curIllegalLocks
IS
SELECT sslock.userid , sslock.planid, sslock.audsid
FROM sslock
WHERE
sslock.audsid NOT IN (SELECT audsid FROM v$session) ;


BEGIN

FOR recIllegalLocks IN curIllegalLocks
LOOP

DELETE FROM sslock WHERE userid = recIllegalLocks.userid AND planid = recIllegalLocks.planid AND audsid = recIllegalLocks.audsid;

END LOOP ;

COMMIT;


END removeIllegalLocks;

END pkg_sslock;
Viewing all 13329 articles
Browse latest View live