tagged [precision]
How do I interpret precision and scale of a number in a database?
How do I interpret precision and scale of a number in a database? I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the colu...
Dividing two integers to produce a float result
Dividing two integers to produce a float result > [Why can't I return a double from two ints being divided](https://stackoverflow.com/questions/7571326/why-cant-i-return-a-double-from-two-ints-being-...
- Modified
- 23 May at 11:47
C# double to decimal precision loss
C# double to decimal precision loss I have a double `"138630.78380386264"` and I want to convert it to a decimal, however when I do so I do it either by casting or by using `Convert.ToDecimal()` and I...
Firing events at microsecond resolution for midi sequencer
Firing events at microsecond resolution for midi sequencer Is there a way to fire events in C# at a resolution of a few microseconds? I am building a MIDI sequencer, and it requires an event to be fir...
C# Converting 20 digit precision double to string and back again
C# Converting 20 digit precision double to string and back again In C#. I have a double (which I've extracted from a database) that has 20 digit precision. In Visual Studio (using QuickWatch) I can se...
Controlling number of decimal digits in print output in R
Controlling number of decimal digits in print output in R There is an option in R to get control over digit display. For example: is supposed to give the calculation results in 10 digits till the end ...
- Modified
- 31 Jan at 08:25
Printf width specifier to maintain precision of floating-point value
Printf width specifier to maintain precision of floating-point value Is there a `printf` width specifier which can be applied to a floating point specifier that would automatically format the output t...
- Modified
- 23 May at 12:2
C# Unable to cast object of type 'System.Double' to type 'System.Single'
C# Unable to cast object of type 'System.Double' to type 'System.Single' before judging that this question is already answered, please read the description. I have this simple code below: ``` Dictiona...
- Modified
- 25 Jan at 13:27
Simulate tearing a double in C#
Simulate tearing a double in C# I'm running on a 32-bit machine and I'm able to confirm that long values can tear using the following code snippet which hits very quickly. ``` static void TestTearingL...
- Modified
- 29 Jan at 13:20
Auto-Interval precision in MS Chart
Auto-Interval precision in MS Chart I'm currently using the charting within .NET using `System.Windows.Forms.DataVisualization.Charting.Chart`. Thus far it seems very powerful, and works great. Howeve...
- Modified
- 7 Oct at 18:5
Is there a way to format a C# double exactly?
Is there a way to format a C# double exactly? Is there a way to get a string showing the value of a `double`, with the decimal places needed to represent its precise value in base 10? For example (via...
- Modified
- 5 Mar at 14:15
Is floating-point math consistent in C#? Can it be?
Is floating-point math consistent in C#? Can it be? No, this is not another question. I've been reading about floating-points a lot lately; specifically, how the on different architectures or optimiza...
- Modified
- 13 Apr at 12:18
Changes to Math.Exp or double implementation in .net 4.5.2
Changes to Math.Exp or double implementation in .net 4.5.2 If I run the statement on a machine with .net 4.5.1 installed, then I get the answer However, if I run the same command on a machine with .ne...
- Modified
- 6 Nov at 10:14
Why does adding double.epsilon to a value result in the same value, perfectly equal?
Why does adding double.epsilon to a value result in the same value, perfectly equal? I have a unit test, testing boundaries: ``` [TestMethod] [ExpectedException(typeof(ArgumentOutOfRangeException))] p...
- Modified
- 17 Dec at 09:27
Rounding mismatch between ASP .net C# Decimal to Java Double
Rounding mismatch between ASP .net C# Decimal to Java Double I am translating .NET code to Java and ran into precision not matching issue. .NET code: ``` private decimal roundToPrecision(decimal numbe...
What is the sum of the digits of the number 2^1000?
What is the sum of the digits of the number 2^1000? This is a [problem](http://projecteuler.net/problem=16) from [Project Euler](http://projecteuler.net/), and this question includes some source code,...
- Modified
- 11 Oct at 04:35