Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Friday, 5 October 2012

Velis auto brightness changelog

4.65.234 (09.09.2017)

  • Fix for crashes in Androids < 5 (sorry)
  • Fix for direct sunlight issues
  • Fix for widget graph getting cleared
  • Fix for Android 8 notification

4.65.233 (27.08.2017) 

  • Fix for very high sensor values (direct sunlight) 
  • Fix for excluded apps sensor value override (high value) 
  • Fix for profile button menu colours

4.65.231 (21.08.2017)

  • Fix for Samsung glitch in excluded apps detection
  • Fix for settings mode
  • Fix for notifications freezing under full sunlight
  • Updated russian translation

4.65.226 (17.07.2017)

  • Updated italian language
  • Top activity check uses much less battery now (Android 5 and above)
  • Major remodel of internal workings. Also less battery usage
  • Setting: brightness update interval: Reduce number of updates which would reduce battery usage
  • Setting: Turn VAB on when screen turns on
  • Setting: enable direct sun screen brightness boost
  • (hopefully) more resilient service (should be stopped by system less often)
  • When change is applied from notification, it's not lost on reboot any more

4.57.215 (09.06.2016)

  • Minor fixes

4.57.214 (21.04.2016)

  • Default camera API back to V1 (V2 gave too much trouble)
  • Navigation drawer always open on wide enough displays

4.56.212 (21.02.2016)

  • Split settings to Basic, Advanced
  • Updated Russian language

4.56.210 (8.2.2016)

  • Fix crashes (wizard, theming)
  • Add support for Android Camera API v2 (might coexist with other camera apps better)
  • Updated Romanian language

4.55.207 (8.1.2016)

  • Fix for excluded apps in 5.0
  • Widget would report tasker content not purchased when it in fact was
  • Other minor fixes

4.55.205 (14.09.2015)

  • Some minor tweaks and fixes

4.55.204 (09.09.2015)

  • FIX: Excluded apps work on Android 5.1 and newer (requires special permission)

4.50.202 (04.9.2015)

  • VAB now works in lock screen on lollipop
  • Better superdim performance
  • Theming

4.21.194 (19.5.2015)

  • FIX: Excluded apps bug

4.21.193 (16.5.2015)

  • FIX: Users on lollipop with camera as sensor will now have brightness for lock screen set to Screen-on sensor value override instead of fixed 80%
  • Added setting for required number of touches (was fixed to 3) to refresh screen brightness when steady screen is on
  • Add setting for brightness when VAB turns off for excluded apps
  • FIX: Theming for buttons
  • FIX: RTL did not work on manually selected RTL language
  • FIX: new app install no longer shows the apply / cancel buttons

4.20.192 (22.4.2015)

  • FIX: Best I can get for lollipop lock screen (sorta works)
    My ultimate (and probably final) attempt at fixing lollipop lock screen. I have researched *everything* I could think of, nothing worked.
    There's no way that I know of that can properly handle this, so I made an extremely ugly hack: I used system AB for those that have it and set screen to 80% brightness for everyone else (those on cameras as sensors). I'm so angry at Google right now and their API breaking changes. :mad:  Total waste of my time, instead of adding semi-cool features, I have to slave around with this.

4.19.191 (21.4.2015)

  • FIX: Wrong overlay type after lock screen (various things didn't work)

4.18.189 (20.4.2015)

  • Yet another try at fixing lollipop lock screen
    • This should also fix issues for delayed camera first read after leaving lock screen
  • FIX: Additional brightness doesn't work
  • FIX: Notification removed from lock screen
  • FIX: Transparent icon removed from lollipop status bar

4.17.188 (15.2.2015)

  • Fix colours

4.16.187 (5.2.2015)

  • Fix unresponsiveness in lollipop

4.15.183 (26.1.2015)

  • Fix flickering lock screen on some devices
  • Fix notification text color

4.13.182 (23.1.2015)

  • Further fixes for lock screen

4.12.181 (20.1.2015)

  • Fix excluded apps in lollipop
  • Fix screen-on responsiveness when Lock screen is NOT among excluded apps
  • Fix screen-on override (sometimes didn't work)
  • Fix non-responsive auto-brightness in some situations
  • Fix crashes at app start

4.10.180 (18.1.2015)

  • Portugese by renatohm
  • Fix lollipop issues
  • Move to iab v3
  • Move to Android Studio
  • All images in app are now vector

4.01.177 (4.6.2014)

  • New translation: greek by pugsang
  • FIX: steadyscreen functionality now much more stable

4.00.175 (7.4.2014)

  • FIX: Notification didn't properly update confirm / cancel button when they were pressed
  • FIX: There were some crashes reported through play store. This fixes them

4.00.168 (6.3.2014)

  • When screen is touched, brightness is not updated for specified time (a new setting - also in wizard)
  • The above setting isn't active when VAB is frontmost app
  • Notification is updated at least every 5 seconds (is now more precise)
  • Unconditional additional brightness
  • Additional brightness accepts negative numbers (will decrease brightness)
  • FIX: Wizard back button now actually works + wizard page count fix
  • FIX: After wizard, settings screen is refreshed to show selected values properly

3.60.166 (24.2.2014)

  • Collapsible settings groups for easier navigation
  • Excluded apps memory and performance improvements
  • FIX: when sensor returned 0, some app functions didn't work correctly
  • FIX: Sometimes changing profiles didn't work correctly

3.60.164 (15.2.2014)

  • Added option to try and reduce erratic sensor readings in some devices (most notably SGS 3 and Note 2)
  • Turning off interactive notification for Android 4.3 and up is now a regular settings option
  • FIX: manual graph edit didn't work for many locales

3.55.162 (11.2.2014)

  • Menu key now opens menu
  • App titlebar now displays that there's a menu to be opened (and opens it)
  • Floating point lux values: much greater precision for low sensor readings
  • Some battery usage optimizations
  • Camera now respects grab interval setting
  • New language: dutch, thanks to Maardiweb

3.55.158 (7.2.2014)

  • Completely redesigned reading interpolator. Less battery usage, fixes a ton of smoothing-related bugs
  • FIX: If service is turned off, cameras will now stop reading
  • FIX: Stopping VAB now completely removes the overlay. Should reduce issues on some devices when VAB was on and still held the brightness

3.55.153 (11.11.2013)

  • Minor visual glitch fixes
  • Interactive notification - only show the buttons once a custom brightness is set
  • No notification option for Android 4.3+
  • Vietnamese by Vu Hien

 3.50.149 (1.11.2013)

  • User interface via fragments

3.15.142 (16.10.2013)

  • A fix for faulty EXIF records in some phone models
  • Chinese language by Jinran Lin

3.15.141 (8.10.2013)

  • Interactive notification
  • Launcher widget can now also render transparent

3.10.135 (23.6.2013)

  • Kernel sensor reader now safer + 0 reading is a valid reading
  • Main screen VAB on/off now properly reflects service state
  • Excluded apps now displays previously selected apps at the top of the list
  • Launcher widget text forced to white

3.08.133 (22.6.2013)

  • FIX: colors in excluded apps list would be strange on some phones
  • Overlay no longer destroyed on screen off to eliminate delay for screen-on
  • SGS 2 now has a kernel sensor reading mode which enables the entire sensor range instead of just 1, 10, 100, 1000

3.05.122 (9.4.2013)

  • When superdimming threshold is set to 100%, it becomes the only method of adjusting screen brightness. Makes brightness adjustments much smoother on some phones (SGS3, GNex 4, etc). Since this method is completely linear, using a brighter preset is recommended or the screen might become too dark.
  • Completely remove brightness window when disabling. Should fix problems with system brightness when VAB is disabled on some phones.
  • Fix graph presets so that graph adjustments at right edge behave normally
  • Fixed strange graph behaviour at graph edges
  • Added graph adjustment sensitivity setting for more precision with graph adjustments
  • Draw superdim threshold in graph (non-user-interactive for now)

3.00.118 (27.3.2013)

  • Fix device exclusions due to camera permission
  • Settings wizard no longer clears saved profiles
    • Settings wizard no longer starts multiple times on new installs
    • Fix sensitivity presets for settings wizard
  • S-curve graph no longer renders beyond graph boundaries
  • Updated translations: slovak

3.00.114 (24.3.2013)

  • Reduced smoothing intervals to reduce flickering on high-end phones
  • Graph period must not be set to 0 (if it's 0, it crashes)
  • Preliminary camera support. See XDA thread for more info

2.82.101 (15.2.2013)

  • Fix for app service disabling itself when a popup is active

2.81.101 (11.2.2013)

  • Custom app language selection
  • Excluded apps (will disable VAB when an excluded app is on top)
  • Super dimming: in addition to backlight also add a screen filter for deeper blacks
  • Separate setting for smoothing time when dimming
  • Android 2.1 & 2.2 compatibility fix (store will still never work below 2.3)

2.81.95 (9.2.2013)

  • Revert to 2.31.84 (13.1.2013) - before screen filter - to resolve some issues
  • All updates since 2.31.84 temporarily disabled
  • Will be added again as they are eliminated as possible causes for the issues detected

2.81.94 (9.2.2013)

  • Android 2.1 & 2.2 compatibility fix (store will still never work below 2.3)
  • Excluded apps list FC fix

2.81.92 (9.2.2013)

  • Fix excluded apps dialog (loads apps smoother)
  • Add superdim activation brightness check
  • Updated translations: persian, slovene

2.80.90 (2.2.2013)

  • Custom app language selection
  • Excluded apps (will disable VAB when an excluded app is on top)
  • Super dimming: in addition to backlight also add a screen filter for deeper blacks
  • Separate setting for smoothing time when dimming
  • Launcher widget 4 slots wide by default

2.31.82 (12.1.2013)

  • Much faster brightness graph rendering (less CPU usage on main screen)
  • Fixed smoothing "hiccups". Graph is now buttery smooth
  • Only use samples after screen-on
  • Add an s-curve graph to the presets
  • Updated translations: slovak, slovene

2.30.80 (6.1.2013)

  • Launcher widget
  • Fix graph adjustments at graph edges
  • Clear settings to default (when wizard starts)

2.20.76 (2.12.2012)

  • Fix for landscape
  • Fix for sensor getting stuck
  • Fix for brightness remaining even when service is disabled

2.20.74 (21.11.2012)

  • App will load custom translation strings (helps translators do their work, see app homepage for instructions)
  • Updated translations: norwegian, italian, slovene
  • Removed requirement for light sensor (will show app to sony xperia users)

2.20.72 (14.11.2012)

  • Settings wizard
  • Service will stop after a minute if auto brightness is disabled
  • Support for proprietary sensor solution on Sony Xperia
  • Better LTR language support
  • Updated translations: german, persian, slovak, spanish

2.01.64 (30.10.2012)

  • Added links to manual and changelog to app menu
  • Screen_on sensor override value (going from light to dark room)
  • Smoothing override setting (when light change is too big, it will just skip to new value)
  • Setting to update system brightness setting
  • Fix for proximity sensor action
  • Fix (hopefully) for graph remaining stuck at some reading
  • Better logging (long-press proximity reading values to set level 0 - 9)
  • Manual graph editing as text
  • Added display of changelog at program start
  • Updated translations: slovene, german
  • Added language: hungarian

2.00.61 (26.10.2012)

  • Fix for in-app billing
  • Updated translations: german

2.00.59 (26.10.2012) - XDA thread only

  • Tasker / locale support
  • In-app billing
  • Updated translations: slovene, german, spanish

 1.20.55 (22.10.2012)

  • A first feeble attempt to prevent service terminations at night
  • Better proximity sensor handling
  • Enabled proguard to make the apk smaller and the code a bit faster
  • Intermediate publish to prepare for tasker and in-app store
  • Added language: russian, japanese, polish
  • Updated translations: slovak, slovene

1.20.49 (15.10.2012)

  • Updated translations: french, norwegian, german
  • Remove strings that won't be used again
  • Fix "track light in graph" buttons appearing without modifying graph
  • Minor fix for about dialog
  • Added language: persian

1.20.46 (13.10.2012)

  • Updated translations for spanish and norwegian
  • Fixed some untranslated strings (new strings for translators)
  • Added about dialog to check current version
  • Applying graph point now eliminates close neighbours and duplicates
  • All new custom settings screen - should be more readable
  • Supported language: french


1.10.40 (9.10.2012)

  • Further optimizations to reduce battery consumption
  • No longer updating android brightness (was just a cosmetic which turned out to use lots of CPU)
  • Reduced last sensor value save interval to one minute to save even more battery
  • Supported languages: slovak, spanish, norwegian (classic and bokmÃ¥l)

 

1.10.37 (8.10.2012)

  • Reduced graph size in landscape mode so that it fits on screen
  • Supported languages: german, lithuanian


1.10.35 (7.10.2012)

  • StartSensors checks for screen off - eliminates sensor time when screen is off
  • Save current sensor value in case of service shutdown - load it when service restarts
  • Change System handler text to "Disable system handler". Previous meaning was just the opposite
  • Choosing a preset or profile now refreshes graph again
  • Supported language: italiano

 

1.10.34 (5.10.2012)

  • Sometimes in low light conditions brightness would drop to minimum without any apparent reason
  • Fix a situation when (un)plugging power won't immediately adjust brightness. 
  • Fix for service activation switch which didn't properly turn off system auto brightness.
  • Fixed class that required Android 2.3 - back to 2.1 support 
  • Translation support
  • Supported language: slovenian

1.10.32 (4.10.2012)

  • No more of no sensor reading
  • Change app description (again) to please competitor

1.10.31 (3.10.2012)

  • Battery usage fix. This should significantly reduce battery usage

1.10.26 (2.10.2012)

  • Fixed sensitivity threshold
  • Removed unnecessary calculations
  • Optimized moving average routine
  • Lower power consumption
  • Change app description to please competitor

1.10.24 (1.10.2012)

  • Fix FC at app first start
  • Fix averaging function

1.10.21 (1.10.2012)

  • Profiles (save the brightness graph under your own selected name)
  • Jitter control to prevent excessive brightness updates
  • Brightness smoothing to make the transitions smoother
  • Minor bug / functionality fixes

1.0.20 (30.9.2012)

  • initial Google play release

1.0.0 (12.8.2012)

  • development start

Friday, 28 September 2012

Velis auto brightness

Get the fun back to your auto brightness...


Free download link: Google play store
Changelog

Strings for translations - all translation help appreciated. The link points to a .zip archive containing all localizations. The base for translations is english text found in "values" directory. Please send the translated strings to my e-mail. Basic knowledge of xml needed: translate only values, not ids. Use plain text editors such as notepad.exe, gedit or kate. Make sure the file is UTF-8 encoded before sending it back.
You can test your translation before sending it to me by placing your translated strings.xml into the phones /sdcard/ directory (root of external storage). The app will detect if such a file is present and load it. Note that currently the menu and some arrays are not translated in this way, but most of the strings should work just fine.

For usage, please see the manual.


Please see XDA homepage about language support. Any translation help most welcome!

Velis auto brightness aims to provide the best possible brightness experience by using your devices sensors to determine the environment you're in. You have complete control over how much brightness will be applied for any given light condition, from selecting the sensors used to fully customizable brightness graph. This is a replacement for system provided auto-brightness functionality usually found in Settings / Display / Brightness

Features:
  • Initial configuration wizard for an easy start
  • User-selectable sensors: light, proximity (cameras planned soon)
  • Supports custom sensors on phones such as Sony Xperia
  • Brightness presets for every taste and screen
  • Profiles (save the brightness graph under your own name)
  • Fully customizable brightness graph to fit your needs or cover your sensor’s faults
  • Extensive sensitivity adjustments (light change threshold, smoothing times up / down, boost threshold)
  • Superdimming to make those shades really dark 
  • Excluded apps (will disable Velis Auto Brightness when they are on top)
  • Launcher widget with on/off button, profile select button and a brightness graph
  • Tasker / Locale support for many sensor readings and app settings
  • Convenient in-app store for premium content (some tasker and widget functionality) and developer support
  • Custom app language
  • Additional brightness when charging
  • Only uses sensors when screen is on conserving battery
Battery consumption notice: This app has been and still is rigorously tested against battery consumption. The only things consuming battery are graph updating (main app screen) and Tasker / Locale conditions for lux, brightness readings (they are frequent events).
However, some phones - most notably some custom ROMs on SGS 2 - will still report this app as the highest battery user even though the battery does not drain any faster than without this app. Please verify your actual battery usage before complaining about the app being listed as a big consumer.

Tasker plug-in / Locale plug-in: Provides conditions for screen on, calculated light reading, proximity sensor reading, calculated brightness. Exposes many settings to locale / tasker for detailed adjustment from these managers. Some of this functionality is premium content available through in-app purchase.

XDA thread (best for support) here.

Disclaimers

Activating multiple auto-brightness applications will not produce expected results so make sure only Velis auto brightness is active while you test it. This file is provided AS IS - without any warranty. The autor will not be liable for any (un)imaginable inconveniences, including, but not limited to your phone frying or your screen starting to shoot launcher widgets at you. :)

LOCALIZATION

This app is available in following languages

* English - base language
* Chinese by Jinran Lin
* Czech by Holly Hell
* Dutch by Maardiweb
* French by Pims83
* Italian by siggey
* German by Chef Koch
* Greek by pugsang
* Hungarian by XT69
* Japanese by Magus
* Lithuanian by cukierkas
* Norwegian by MrMastodon
* Persian by Alireza Afkar
* Polish by trur3
* Portuguese by renatohm\n
* Russian by Pavel Utochkin
* Romanian by rapttorx
* Slovak by kubics
* Spanish by UnderXP
* Vietnamese by Vu Hien
* Slovene

Thanks to all the translators for their hard work.

REQUIRED PERMISSIONS

WRITE_SETTINGS: Required to turn off system auto-brightness. This is for your convenience so that you don't have to go to system settings manually turning off system auto brightness
RECEIVE_BOOT_COMPLETED: Required for starting the service at bootup so that auto brightness actually works without manual program start.
SYSTEM_ALERT_WINDOW: Required for auto brightness to actually operate
GET_TASKS: Needed to detect current top activity for app exclusion list and proper screen filter operation
CAMERA: Determine environment brightness through cameras.
BILLING: Required for in-app store

SCREENSHOTS

Main screen



Settings

 


Notification
 

Launcher widget
 

Saturday, 10 March 2012

Increasing transaction performance and getting rid of deadlocks caused by high transaction count

If you're seriously into database applications, you are sure to have bumped into a customer that was really big. Like so big that their server reached a point where it was no longer capable of serving all the transactions requested by the software / users. Personally reaching this point is a genuine nightmare for me: monday everything is fine, everybody works normally and we are happy. Then they install that damned robot / new warehouse / whatever which increases the transaction count for the system. So tuesday all hell breaks loose: everything people try to do takes ages and more often than not reports the deadlock victim error. Naturally, we are no longer happy.

Solving such situations requires expensive performance tuning software, lots of analysis and optimizing query by query so that you can lift at least some burden off the poor server which is of course still struggling, but *just* not so much that everything would halt any more. To quickly recap your performance tuning options (the real purpose of this article lies beyond this list :-) ):
  1. Indexes
    1. Adding a good clustered index for that most used aggregate query will work wonders usually.
    2. For less important queries add normal indexes. 
    3. But beware: adding too many indexes kills your insert / update performance because the SQL server needs to update too many things for one addition or correction in the database.
    4. Also bear in mind that having single - column indexes will not really help much with multi-column where or group by statements. Sure - it helps a little, but nowhere near what a proper multi - segment index would do for you here
    5. Recent SQL servers also have the capability to extract all the data for a query just from an index. So if you have an (aggregate) query which only uses a few of the table columns, usually creating an index just for this query is not such a bad idea, especially if you're running said query all the time
  2. Locking
    1. You should be very careful what you lock and when
    2. The most common culprit are usually configuration tables - if you store application config in a table. This table is accessed for pretty much anything all the time and moving all these accesses out of transactions or using with(nolock) hints helps tremendously
    3. The next most common lock targets are link tables - the tables that establish data relations. Like this delivery item was originally requested in order #xxx. Depending on application design you will most likely have at least one such table in your system. Also needs special care, but unfortunately can't really be optimized since it must be kept in the transaction for data consistency.
    4. Anyway, you should let the SQL server manage locks at a level as fine as possible so that there are as few as possible lock races
  3. Query optimization
    1. Can't really say much about this since it's query - dependant, but I have seen some really yucky queries in my day
    2. Usually a simple rewrite of a nasty query will decrease query execution time multiple - fold
    3. Take care not to use multiple (basically equal) subqueries - rather use them in a join (and a group by)
    4. Joins are (almost) always better than subqueries
    5. Dynamic views (select from select) are also better to be avoided if possible though sometimes they are the basis for your query
    6. Once you detect a query that needs optimization, you optimize it and make sure indexes in the respective tables cover your query's needs
    7. General rule of thumb is that there should be absolutely no table scans executing the query and possibly no index scans. Scans take time, seeks do not!
    8. If you can't think of a way to optimize a really bad query, drop me a mail. For a modest fee I will do this for ya }:-)) (Seriously though: see the donate button at the bottom of the page and don't write without clicking it)
This is about it. The above procedures will usually give you good results but they take many hours of analysis and optimization and thus take valuable time of your programmers who could be better spending their time creating new (bad) queries for new software functionality.

There are also two programmatic options you have at your disposal which will do a lot more general good with less effort for your programmers - since they affect all transactions ever issued throughout your system. The only prerequisite here is that your application is well designed and you actually have access to code that starts / finishes / aborts transactions:

The first option is to simply catch deadlock errors and restart the transaction automatically. Aside from taking even more time, your user will never know that a deadlock even occurred. However, the SQL server will know since this will put even more stress on it - and don't forget: you're having the deadlocks because you placed too much stress on the server in the first place.

The second option is to serialize your transactions. This one is the purpose of this article. You see, until yesterday we at the company I work for always went for performance tuning / optimization. However yesterday I developed a really simple solution which I didn't really believe would solve anything. I just hoped it would do *some* good for the poor server at our client's. Boy was I in for a treat :-)
I was at the client's two days ago and I can testify that their server was just way too loaded. I have personally started 10 transactions. Of those, 5 went straight through in less than a second (normal time), 3 deadlocked after more than 30 seconds (definitely bad) and 2 went through in more than 30 seconds (meaning somebody else probably got the deadlock). Disk time was at 100% all the time and CPU was only low because the CPU was waiting for the disks. All that on one of the most powerful database servers I have ever seen deployed at our customers. The situation was just horrible. All because we installed some automatic data transformations to support their new huge automatic warehouse and a few of packaging robots.
Once I installed the serialization code, I first thought the entire system crashed. There was no activity for two minutes or so. Only after this initial period where server was so stumped that it simply couldn't do anything, things started rolling:Transactions started to come in in hundreds per minute as all the automated jobs started doing their work that failed in their previous attempts due to deadlocks. For the first fifteen minutes all I could look at was that flood of transactions being mostly nicely serialized. But then everything just stopped again - revealing that there are actually only about 20 transactions per minute. Once I reached this state, the server was at practically 0% CPU usage and 0% disk time - just smooth sailing from now on. Total, entire and unyielding success!!!!
So it turns out that it's not the server that is too weak. It's just that SQL server tries to accomodate all users by running multiple transactions in parallel, but that leads to locking and lock waiting and transactions that should take milliseconds start taking seconds. During that time new transactions come worsening the situation further - until the entire thing slows down to a deadlock infested crawl.

OK, enough of blabbing, here's what I did:
The first prerequisite was that I needed to add some code just before calls to start / stop transactions. All that code does is to call a stored procedure on the SQL server just before it initiates a transaction and again calls the same procedure just after it finishes / aborts the transaction. This is super important: doing this inside a transaction would just kill the purpose because the locking would be done on the scheduling table as well.
Anyway, really simple, really easy. The stored procedure is designed so that it will return a "transaction ID" as well as "permission" to start transaction. The "permission" part of the result is actually number of transactions that will be allowed to start before this one so you can even notify the user while this particular transaction is waiting to be started.
The sample function itself has three possible input value forms:
  1. "Announcement" null - I want to start a new transaction, give me an ID and queue #. When Queue # reaches 0 I can start the transaction
  2. "Is it my turn yet?" transaction ID (returned by previous anouncement call) - I'm waiting to be allowed to start. What's my queue #?
  3. "I'm done" negative transaction ID - I'm finishing / aborting transaction

Of course you can do this any way you like, my particular implementation is done like described, but it's in no way perfect. That's the beauty of this method: you can make your transaction scheduler any way you want as long as you serialize at least some transactions.

The general concept of this solution is this:
First we assume that all transactions are equal and that they should be executed in order (as opposed to simultaneously) to lessen the locking stress on the server. Then we eliminate any transactions that appear dead and all transactions that are classified as long taking. The method assumes that most transactions are very short and attempts to serialize those while it leaves the user-prompt infested ones take as long as they want. As explained above, it turns out that the locking stress really is stressful on the server. I could not believe the results when I saw what was going on, but the final results show that serializing the fast transactions helps more than just a lot.

A more detailed description:
Before starting a transaction, we add its announcement into a special table. Since we're doing this outside the transaction, the table is available to all at all times. A stored procedure makes sure that the entire thing runs on the server and is fast because of this. I'm running the second form (is it my turn yet?) every 100 milliseconds (on each client) and have observed no ill effects whatsoever. In order to keep the entire thing fast, I remove all records older than 15 minutes so as to keep the table size as small as possible.
The stored procedure makes sure the transaction is inserted into the queue (announcement), checks current queue # (announcement and is it my turn yet) and marks transaction as still alive and also marks the transaction as finished when called with negative transaction ID (I'm done).
Just in case some transactions take too long, the procedure will ignore them and allow other transactions to run in parallel and also if one transaction is no longer checked upon, it will also be ignored. This approach makes sure that a single transaction can't be blocking the entire system just because it's slow - small transactions just run and hope that the large transaction won't block them (turns out this is extremely rarely the case).
The client registers the transaction using the first call form and then waits in a loop until the Queue # drops to 0 for the transaction. Then transaction is ran as it always was and when it is finished, the client also reports that it is finished.

So if you now already modified your database access library to include stored procedure call as described above, all that is missing is the stored procedure and related call. You should now be running the new application just to see that nothing changed. Once the stored procedure is added, the magic should begin. I have included the two statistics selects at the bottom just for the nice side-benefit (transaction statistics) :-)
This solution is in MS SQL T-SQL language, but I suppose a conversion to any other database should be easy.

The helper table:
create table transaction_serializer (
  id int identity(1,1) primary key,
  time_inserted datetime null,
  time_queried  datetime null,
  time_executed datetime null,
  time_finished datetime null
)
create index transaction_serializer_i1 on transaction_serializer (time_finished, id)
create index transaction_serializer_i2 on transaction_serializer (time_queried)

The stored procedure:
create procedure serialize_transaction(@transid int)
as begin 
  set nocount on
  set ansi_warnings off
  declare @transok int

  --Remove 15 minute old records to keep this scheduler table small and fast
  delete from transaction_serializer where time_inserted < dateadd(mi, -15, getdate())
  if @transid < 0 begin
    --Transaction done
    update transaction_serializer set time_finished = GETDATE() where ID = -@transid
    set @transok = 0
  end
  else begin
    if @transid is null
    begin
      -New transaction
      insert into transaction_serializer (time_inserted) values(GETDATE())
      set @transid = @@IDENTITY
    end
    -Determine queue# for transaction
    select @transok = COUNT(*)
      from transaction_serializer
     where time_finished is null and id < @transid --How many transactions in queue before ours
       and isnull(datediff(ms, time_executed, GETDATE()),0) < 2000 --Eliminate long taking transactions
       and isnull(datediff(ms, time_queried,  GETDATE()),0) < 2000 --Eliminate seemingly dead transactions

    if @transok = 0 begin
      --This transaction first, let's start it
      update transaction_serializer set time_executed = GETDATE(), time_queried  = GETDATE() where ID = @transid
    end
    else begin
      --Just mark the transaction still alive
      update transaction_serializer set time_queried  = GETDATE() where ID = @transid
    end
  end 
  --transid contains transaction id which the client then uses for querying queue # and marking transaction as done
  --transok is 0 - if the transaction can be started or positive number which tells us that our transaction is still waiting
  select @transid, @transok
end

The additional selects to retrieve statistics:
select *, datediff(ms, time_executed, time_finished), datediff(ms, time_inserted, time_executed) from transaction_serializer
select count(*) as NumTrans, 
       max(datediff(ms, time_executed, time_finished)) as MaxTransTime, 
       avg(datediff(ms, time_executed, time_finished)) as AvgTransTime, 
       sum(case when datediff(ms, time_executed, time_finished) > 2000 then 1 else 0 end) as NumOfLongTransactions,
       max(datediff(ms, time_inserted, time_executed)) as MaxWaitTime,
       max(datediff(ms, time_inserted, time_executed)) as AvgWaitTime
  from transaction_serializer


Now you know that implementing this just saved your ass with the client. So for your convenience you can express your gratitude by clicking the following button. Don't be shy, you know I just saved your company thousands in costs of performance tuning :-) In case you're feeling stingy at least click a link in the banner on the right side of this page.

  For possible further inquiries drop me a mail at jure dot erznoznik at gmail dot com.

 

Friday, 3 February 2012

Declaring and using C++ properties like in Delphi or C#

I find properties a marvelous concept. You gain so much more control over class members not to mention declaration of the stuff places everything in just a few lines of code. It is immediately visible just by looking at class declaration what the class offers you and what it doesn't.
Anybody who switched from Delphi to C++ will know what I'm talking about.

I have been looking for a way to support properties in C++ for a while now without much luck. Looking at the implementations found on the net, there are two types of implementation:
  1. Property class that stores getter / setter / lvalue pointers
  2. subclasses with pre-processor macro-style implementation
Each of the methods used has its merits and shortfalls. I personally don't like the first approach because it means that for each property declared you waste 3x4 bytes (32 bit) or even 3x8 bytes (64 bit). Plus there's the initialization code setting those pointers adding just about twice the memory consumption on top of the pointers allocated. And I don't like losing so much memory for something as simple as that.
The second approach is a lot more complex to nail down, but once you have the macros, usage is just as easy as the first approach and there is only one detail you need to watch out for: empty class members in C++ take 1 byte of storage (by standard) just for indexing purposes. You can gain this one byte back simply by storing the property l-value inside.

My original intent was to get this thing as far as Delphi's TObject was, but I fell short of implementing the storage / identification subsections. This way you can't do the following two things:
  1. myInstance.setProperty("propertyname", value)
  2. stream << myInstance, stream >> myInstance
So if you feel up to it, I'm all inbox for ya :) I will even explain how it can be done further down.

As explained I decided for the second approach with macros and subclasses because it is much more efficient memory usage-wize. Originally it was intended for there to be only one macro which would do it all, but it turns out C++ preprocessor is pretty dumb and doesn't really allow you to do such things. So I had to split the code in multiple macros, the differences in usage being explained around line 80. Please forgive me for the messed up code, this really wasn't easy to achieve.

At the end of the code there's also the TObject class declared showing most of the functionality off. You can use that class as an example for your usage or use it as the base class for your library. It's up to you.

As mentioned above, I failed to implement storage / load. Turns out my C++ still isn't up to par though this is the most complete property implementation I know of in the net. Only Qt library manages to surpass this implementation, but they have their own preprocessor. Anyway, this code will compile both on MS VC and gcc.

In order to implement that, a global std::map would ned to be declared into which properties would register themselves in their constructors. The key would be class + property and data would be pointers to getter, setter, default and store method pointers. Additionally, each property type would need conversion to string done in order to be able to store classes into XML or something. Having all this implemented, it is then trivial to write getProperty and setProperty functions as well as properly implement saveToXML / loadFromXML.

Anyway, I still feel this implementation of C++ property support is pretty awesome in comparison to other attempts and it should be posted to the almighty net for your pleasure under modified BSD license. The code compiles under MS VC++ and gcc. Have fun.


#include <string>
#include <sstream>
#include <iostream>
#include <typeinfo>
#include <list>

typedef std::wstring vString;

//Support macros begin
//getter macros
#define PROPERTY_GETTER(getter, prop_type, prop_my_struct) \
operator prop_type() { return getter; }                    \
friend std::wostream& operator << (std::wostream &out, prop_my_struct aValue) { out << (prop_type)aValue; return out; }

#define V_GETTER(varName)           getBase()->varName
#define F_GETTER(funcName)          getBase()->funcName()
#define S_GETTER(varName)           varName
#define T_GETTER(varName, funcName) funcName
#define N_GETTER 

//Setter macros
#define PROPERTY_SETTER(setter, prop_type, param_type, prop_my_struct, streamcode) \
prop_my_struct operator = (param_type aValue) { setter; return *this; }            \
friend std::wistream& operator >> (std::wistream &in, prop_my_struct aValue) { streamcode return in; }

#define V_SETTER(varName)  getBase()->varName = aValue
#define F_SETTER(funcName) getBase()->funcName(aValue)
#define S_SETTER(varName)  varName = aValue
#define T_SETTER(funcName) funcName(aValue)
#define N_SETTER 

//Default values
#define DEFAULT_EXPAND(prop_type, value) prop_type(value)
#define N_DEFAULT
//Internal storage (saves one byte for struct default allocation size)
//use N_STORE if you have external storage or no storage
#define STORAGE_EXPAND(prop_type, value) private: prop_type value; public:
#define N_STORE __intstore_func() {}
#define STORAGE_STD(type, defaultval, store)                                   \
bool isDefault() { return (type)(*this) == DEFAULT_EXPAND(type, defaultval); } \
bool stored() { return store; }
#define STORAGE_STDC(type, defaultval, store)      \
bool isDefault() { return (type)(*this) == NULL; } \
bool stored() { return store; }
#define STORAGE_STDS(type, defaultval, store) \
bool isDefault() { return value == NULL; }    \
bool stored() { return store; }

//Constructor / destructor
#define PROPERTY_CTOR(name, set_default) property_##name##_class() { getBase()->registerProperty(L#name); set_default}
#define PROPERTY_DTOR(name, del_obj) ~property_##name##_class() { del_obj }

#define PROPERTY_ALL(base, name, type, ctor, dtor, storage, getfunc, setfunc, defstore, addcode)  \
struct property_##name##_class                                                        \
{                                                                                     \
  ctor                                                                                \
  dtor                                                                                \
  storage                                                                             \
  getfunc                                                                             \
  setfunc                                                                             \
  base *getBase() { return ((base *) ((char *) this - offsetof( base, name ))); }     \
  defstore                                                                            \
  friend class base;                                                                  \
private:                                                                              \
  addcode                                                                             \
} name
//Support macros end

/*
  Note that I had to do some compromises because C++ preprocessor is so dumb
  I found it impossible to support various data types in one macro, so I had to use multiple
  The initial letter before PROPERTY designates a particular data type I supported with that macro
   PROPERTY - standard ordinal type property
  CPROPERTY - property that supports class data types. The macro will support -> operator which 
              accesses class members directly
  SPROPERTY - same as CPROPERTY, but manages allocation of the value (class instance)
  IPROPERTY - supports container types

  The letter trailing PROPERTY designates the use of getter / setter
  _PROPERTY  - property with both a getter and a setter
  _PROPERTYG - property with just a getter (read-only property)
  _PROPERTYS - property with just a setter (write-only property)
*/

//Standard basic type properties
#define PROPERTY(base, name, type, storage, getfunc, setfunc, defaultval, store, addcode)  \
PROPERTY_ALL(base, name, type,                                                             \
             PROPERTY_CTOR(name, ),                                                        \
             PROPERTY_DTOR(name, ),                                                        \
             STORAGE_EXPAND(type, storage),                                                \
             PROPERTY_GETTER(getfunc, type, property_##name##_class &),                    \
             PROPERTY_SETTER(setfunc, type, const type &, property_##name##_class &, in >> aValue;), \
             STORAGE_STD(type, defaultval, store), addcode)

#define PROPERTYG(base, name, type, storage, getfunc, addcode)                             \
PROPERTY_ALL(base, name, type,                                                             \
             PROPERTY_CTOR(name, ),                                                        \
             PROPERTY_DTOR(name, ),                                                        \
             STORAGE_EXPAND(type, storage),                                                \
             PROPERTY_GETTER(getfunc, type, property_##name##_class &),                    \
             ,, addcode)

#define PROPERTYS(base, name, type, storage, setfunc, addcode)                             \
PROPERTY_ALL(base, name, type,                                                             \
             PROPERTY_CTOR(name, ),                                                        \
             PROPERTY_DTOR(name, ),                                                        \
             STORAGE_EXPAND(type, storage),                                                \
             ,                                                                             \
             PROPERTY_SETTER(setfunc, type, const type &, property_##name##_class &, in >> aValue;), \
             , addcode)

//class / struct type properties (note that properties can only be pointers to actual l-values)
//!!!Note that below code assumes that CPROPERTY has storage :( Since I don't know better I may have to re-declare the below 3 macros such that I have one set for initialized storage and one for no storage
#define CPROPERTY(base, name, type, storage, getfunc, setfunc, store, addcode)                  \
PROPERTY_ALL(base, name, type *,                                                                \
             PROPERTY_CTOR(name, storage = NULL;),                                              \
             PROPERTY_DTOR(name, ),                                                             \
             STORAGE_EXPAND(type *, storage),                                                   \
             PROPERTY_GETTER(getfunc, type *, property_##name##_class)                          \
             type * operator ->() { return (type *)(*this); },                                  \
             PROPERTY_SETTER(setfunc, type *, type *, property_##name##_class, in >> aValue;),  \
             STORAGE_STDC(type *, NULL, store), addcode)

#define CPROPERTYG(base, name, type, storage, getfunc, addcode)                                \
PROPERTY_ALL(base, name, type *,                                                               \
             PROPERTY_CTOR(name, ),                                                            \
             PROPERTY_DTOR(name, ),                                                            \
             STORAGE_EXPAND(type *, storage),                                                  \
             PROPERTY_GETTER(getfunc, type *, property_##name##_class)                         \
             type * operator ->() { return (type *)(*this); },                                 \
             ,, addcode)

#define CPROPERTYS(base, name, type, storage, setfunc, addcode)                                \
PROPERTY_ALL(base, name, type *,                                                               \
             PROPERTY_CTOR(name, ),                                                            \
             PROPERTY_DTOR(name, ),                                                            \
             STORAGE_EXPAND(type *, storage),                                                  \
             ,                                                                                 \
             PROPERTY_SETTER(setfunc, type *, type *, property_##name##_class, in >> aValue;),, addcode)

#define SGET_FUNC(type) *(value == NULL?value = new type():value)
#define SSET_FUNC(type) if (value == NULL) value = new type(aValue); else *value = aValue
#define SPROPERTY(base, name, type, store, addcode)                          \
PROPERTY_ALL(base, name, type *,                                             \
             PROPERTY_CTOR(name, value = NULL;),                             \
             PROPERTY_DTOR(name, if (value != NULL) delete value;),          \
             STORAGE_EXPAND(type *, value),                                  \
             PROPERTY_GETTER(SGET_FUNC(type), type, property_##name##_class &) \
             type operator ->() { return (type)(*this); },                   \
             PROPERTY_SETTER(SSET_FUNC(type), type, type, property_##name##_class &, in >> aValue;), \
             STORAGE_STDS(type, NULL, store), addcode)


//indexed properties that would map to a get() or set() function (but not both) are impossible
//  since both [] and () operators can only be overloaded such that they return reference to the element in question
//  this would mean you actually have to have data behind the get() / set() function, which is not the purpose 
//  of such properties thus I decided not to support indexed properties! :(
//  instead I support:
//Container (array) properties
//  These are simplified class properties that store the container pointer. The pointer is allocated only upon access
//  Read only access is supported (can't assign a different instance to the property)
//  To avoid unnecessary allocation, a bool empty() method is provided which returns whether the object is instantiated
//  This is done so because std::map takes 28 bytes and std::list takes 24 bytes in 32-bit MSVC
//  I prefer 4 (32 bit) / 8 (64 bit) bytes of fixed storage per class instance
//  These properties are always read/write though meaning you can access ALL container methods / operators once instantiated. Sorry :)
//  In order to be able to load / store these properties (main intent for their support), you must supply the following:
//    begin() and end() iterator methods as implemented in std:: container classes
//    << and >> operator overloads for the container types
#define AGET_FUNC(type) (value == NULL?value = new type():value)
#define ASET_FUNC(type) if (value == NULL) value = new type(); value = aValue
#define ASTORAGE(store)            \
bool isDefault() { return false; } \
bool stored() { return store; }

#define IPROPERTY(base, name, type, store, addcode)                                               \
PROPERTY_ALL(base, name, type *,                                                                  \
             PROPERTY_CTOR(name, value = NULL;),                                                  \
             PROPERTY_DTOR(name, if (value != NULL) delete value; ),                              \
             STORAGE_EXPAND(type *, value),                                                       \
             PROPERTY_GETTER(AGET_FUNC(type), type *, property_##name##_class)                    \
             type * operator ->() { return (type *)(*this); },                                    \
             PROPERTY_SETTER(ASET_FUNC(type), type *, type *, property_##name##_class, in >> aValue;),  \
             ASTORAGE(store), addcode)

class Object
{
public:
  Object() {}
  Object(const Object &aValue) {}
  virtual ~Object() { parent = NULL; } 

  virtual Object *create() const = 0; //"virtual default constructor"
  Object *create(Object *aParent) { Object *res = create(); res->parent = aParent; return res; } 
  virtual Object *copy() { Object *obj = create(); obj->copyMembers(); return obj; } //"virtual" copy constructor
  virtual void copyMembers() {}

   PROPERTYG(Object, className, vString, N_STORE, F_GETTER(getClassName),);
  SPROPERTY (Object, name,      vString, true, );
  CPROPERTY (Object, parent,    Object,  mParent, S_GETTER(mParent), F_SETTER(setParent), false,);
  IPROPERTY (Object, children,  std::list<Object *>, true,);
   PROPERTY (Object, tag,       int, value, S_GETTER(value),   S_SETTER(value), 0, true, ); //for some reason MSVC 2008 increases the object by 20 bytes if I use an int64 here :(

protected:
  vString getClassName() 
  {
    std::string s(typeid(*this).name());
    return vString(s.begin(), s.end()); 
  }

  void children_append(const Object *aParent);

  void setParent(Object *aParent)
  {
    //First remove any existing parent
    if (parent != NULL)
      parent->children->remove(this);
    parent.mParent = aParent;
    //Add me to the new parent's children list
    if (aParent != NULL)
      parent->children->push_back(this);
  }

  static void registerProperty(vString name) {} //TODO: Do the code, man
  static void registerClass() {} //TODO: Do the code, man

  vString getProperty(vString aPropertyName) {} //TODO: Do the code, man
  void setProperty(vString aPropertyName) {} //TODO: Do the code, man

  friend std::wostream& operator << (std::wostream &out, Object &aValue) { /*out << aValue;*/ return out; }
  friend std::wistream& operator >> (std::wistream &in,  Object &aValue) {  }
};