# tagged [precision]

#### How do I interpret precision and scale of a number in a database?

How do I interpret precision and scale of a number in a database? I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the colu...

#### Dividing two integers to produce a float result

Dividing two integers to produce a float result > [Why can't I return a double from two ints being divided](https://stackoverflow.com/questions/7571326/why-cant-i-return-a-double-from-two-ints-being-...

- Modified
- 23 May at 11:47

#### C# double to decimal precision loss

C# double to decimal precision loss I have a double `"138630.78380386264"` and I want to convert it to a decimal, however when I do so I do it either by casting or by using `Convert.ToDecimal()` and I...

#### Firing events at microsecond resolution for midi sequencer

Firing events at microsecond resolution for midi sequencer Is there a way to fire events in C# at a resolution of a few microseconds? I am building a MIDI sequencer, and it requires an event to be fir...

#### C# Converting 20 digit precision double to string and back again

C# Converting 20 digit precision double to string and back again In C#. I have a double (which I've extracted from a database) that has 20 digit precision. In Visual Studio (using QuickWatch) I can se...

#### Controlling number of decimal digits in print output in R

Controlling number of decimal digits in print output in R There is an option in R to get control over digit display. For example: is supposed to give the calculation results in 10 digits till the end ...

- Modified
- 31 Jan at 08:25

#### Printf width specifier to maintain precision of floating-point value

Printf width specifier to maintain precision of floating-point value Is there a `printf` width specifier which can be applied to a floating point specifier that would automatically format the output t...

- Modified
- 23 May at 12:2

#### C# Unable to cast object of type 'System.Double' to type 'System.Single'

C# Unable to cast object of type 'System.Double' to type 'System.Single' before judging that this question is already answered, please read the description. I have this simple code below: ``` Dictiona...

- Modified
- 25 Jan at 13:27

#### Simulate tearing a double in C#

Simulate tearing a double in C# I'm running on a 32-bit machine and I'm able to confirm that long values can tear using the following code snippet which hits very quickly. ``` static void TestTearingL...

- Modified
- 29 Jan at 13:20

#### Auto-Interval precision in MS Chart

Auto-Interval precision in MS Chart I'm currently using the charting within .NET using `System.Windows.Forms.DataVisualization.Charting.Chart`. Thus far it seems very powerful, and works great. Howeve...

- Modified
- 7 Oct at 18:5

#### Is there a way to format a C# double exactly?

Is there a way to format a C# double exactly? Is there a way to get a string showing the value of a `double`, with the decimal places needed to represent its precise value in base 10? For example (via...

- Modified
- 5 Mar at 14:15

#### Is floating-point math consistent in C#? Can it be?

Is floating-point math consistent in C#? Can it be? No, this is not another question. I've been reading about floating-points a lot lately; specifically, how the on different architectures or optimiza...

- Modified
- 13 Apr at 12:18

#### Changes to Math.Exp or double implementation in .net 4.5.2

Changes to Math.Exp or double implementation in .net 4.5.2 If I run the statement on a machine with .net 4.5.1 installed, then I get the answer However, if I run the same command on a machine with .ne...

- Modified
- 6 Nov at 10:14

#### Why does adding double.epsilon to a value result in the same value, perfectly equal?

Why does adding double.epsilon to a value result in the same value, perfectly equal? I have a unit test, testing boundaries: ``` [TestMethod] [ExpectedException(typeof(ArgumentOutOfRangeException))] p...

- Modified
- 17 Dec at 09:27

#### Rounding mismatch between ASP .net C# Decimal to Java Double

Rounding mismatch between ASP .net C# Decimal to Java Double I am translating .NET code to Java and ran into precision not matching issue. .NET code: ``` private decimal roundToPrecision(decimal numbe...

#### What is the sum of the digits of the number 2^1000?

What is the sum of the digits of the number 2^1000? This is a [problem](http://projecteuler.net/problem=16) from [Project Euler](http://projecteuler.net/), and this question includes some source code,...

- Modified
- 11 Oct at 04:35