- Is relational data the main reason to start LINQ development?

It is a common misconception that LINQ is strongly coupled to relational data. If you look at the various papers on my homepage (http://research.microsoft.com/en-us/um/people/emeijer/ErikMeijer.html) you see that I emphasize LINQ’s monadic roots. Monads are a way to reason about general “computations” and querying a relational database is just one possible example of that. Take Rx http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx for instance, you see that we define LINQ over event-based and asynchronous computations. That is quite different from your typical relational database I would say. Other examples include LINQ to Sharepoint, LINQ to Twitter, …. Those query “data” but not relational data. Have a look at http://homepages.inf.ed.ac.uk/wadler/topics/monads.html for more examples of possible LINQ data sources.

Если выкинуть работу с реляционным источником из списка, то вся обозначенная проблема становится достаточно несущественной и незначительной, чтобы не расшибаться в лепешку ради специальной поддержки специального механизма в языке.

- "if you throw away relational data application then impedance mismatch problem becomes somewhat inessential and insignificant to put such a huge effort to add a special support into the language". Do you think it is correct?

As the answer to the first question implies, LINQ is not tied to relational data. What I am trying to say in the interview you quote is that what LINQ does is to emphasize the correspondence between different data models by defining a common interface/pattern that a “data source” can expose and then define a query language in terms of that interface. This in contrast to ther approaches where people are converting their data into a universal data model and querying on that.
LINQ is about composing computations, querying relational data is one special, and important, form of composing computations.

- What are the main advantages of LINQ over using plain simple loops when LINQ is not used for relational data interaction?

That is not a correct question since, again, you assume that you are dealing with explicit data sources. How could you use loops to do Rx-style computations?

- Do you think LINQ-like approach would be a good addition for other languages?

Well, F#, Haskell and Scala all have (monad) comprehensions, so obviously yes.

G>Я спрашиваю: "дальше что", то есть, ты-то что думаешь об этих примерах сам? Вот честно? Например, ты можешь сказать что-то вроде "да, действительно, вы подобрали поганые примеры, эти примеры не раскрывают силу технологии, она сильна в другом".

Вышли из леса суровые русские мужики и увидели японскую бензопилу.
— А-а, б**! — сказали суровые русские мужики и засунули в пилу щепку.
— Вжик! — сказала пила и распилила щепку.
— У-у, б**! — сказали суровые русские мужики и засунули в пилу доску.
— Вжжик! — сказала пила и распилила доску.
— У-у, б**! — сказали суровые русские мужики и засунули в пилу бревно.
— Вжжжжик! — сказала пила и распилила бревно.
— У-у, б**!!! — сказали суровые русские мужики и засунули в пилу рельс.
— Хррр-дзинь — сказала пила и сломалась.
— А-а, б**... — сказали суровые русские мужики и пошли валить лес топорами.

const = 5

G>>И как мне это надо "по-правильному" понимать?

Yet, I was not really setting out to create a whole full-blown ORM system. I just needed something to kick the tires of LINQ with, a ‘mock’ LINQ provider if you will.

    static Dictionary<string, List<Percent>> CalcPercents2(
        ProvidingVolume[] providingVolumes, ReceivingVolume[] receivingVolumes)
    {
      Dictionary<string, List<ProvidingVolume>> providingGroup = new Dictionary<string, List<ProvidingVolume>>();
      foreach (ProvidingVolume pv in providingVolumes)
        providingGroup.GetOrCreateValue(pv.Provider).Add(pv);

      Dictionary<string, List<ReceivingVolume>> receivingGroup = new Dictionary<string, List<ReceivingVolume>>();
      foreach (ReceivingVolume rv in receivingVolumes)
        receivingGroup.GetOrCreateValue(rv.Account).Add(rv);

      Dictionary<string, double> receivingSum = new Dictionary<string, double>();
      foreach (List<ReceivingVolume> list in receivingGroup.Values)
        receivingSum[list[0].Account] = list.Sum(delegate(ReceivingVolume rv) { return rv.Volume; });

      Dictionary<string, List<Percent>> percents = new Dictionary<string, List<Percent>>();
      foreach (List<ProvidingVolume> list in providingGroup.Values)
      {
        double sum = list.Sum(delegate(ProvidingVolume pv) { return pv.Volume; });

        foreach (ProvidingVolume pv in list)
        {
          List<ReceivingVolume> rv = receivingGroup.FindObject(pv.Account);
          if (rv != null)
          {
            foreach (ReceivingVolume r in rv)
            {
              Percent percent = new Percent();
              percent.Provider = pv.Provider;
              percent.Receiver = r.Receiver;
              percent.Account = pv.Account;
              percent.Value = pv.Volume / sum * r.Volume / receivingSum[r.Account];

              percents.GetOrCreateValue(percent.Provider).Add(percent);
            }
          }
        }
      }

      return percents;
    }

        static Dictionary<string, List<Percent>> CalcPercents(
            ProvidingVolume[] providingVolumes, ReceivingVolume[] receivingVolumes)
        {
            var percents = 
                from percent in
                    from prov in
                        from p in providingVolumes
                        group p by p.Provider into pg
                        let total = pg.Sum(v => v.Volume)
                        from pp in pg
                        select new { pp, total }
                    join rec in
                        from r in receivingVolumes
                        group r by r.Account into rg
                        let total = rg.Sum(v => v.Volume)
                        from rr in rg
                        select new { rr, total }
                    on prov.pp.Account equals rec.rr.Account
                    select new Percent
                    {
                        Provider = prov.pp.Provider,
                        Receiver = rec.rr.Receiver,
                        Account  = prov.pp.Account,
                        Value    = prov.pp.Volume / prov.total * rec.rr.Volume / rec.total
                    }
                group percent by percent.Provider;

            return percents.ToDictionary(p => p.Key, p => p.ToList());
        }

int lastNdx = int.MinValue;
var q1 = Enumerable.Range(1, arr.Length - 1).Where(i => {
    if (i == lastNdx + 1 || arr[i] != arr[i - 1] + 1) {
        return false;
    } else {
        lastNdx = i;
        return true;
    }
}).Select(i => arr[i]);

var q2 = arr.Where(n => q1.Contains(n - 1) || q1.Contains(n + 1));

The idea of LINQ is to solve the "impedance mismatch" between the different data models. Look at the 3 prevalent data models: Objects, XML and Relational Data. Programmers have to struggle with that daily when they write programs, mess around, querying relational data, doing some computations, then exposing the XML, and then vice versa. This is a problem that I wanted to attack when I moved from Academia to industry. There are several ways people try to attack this problem traditionally, and one is by saying: "Let's take one of this data models and take them as the uber model". You can say one popular view is XML, and say "Oh well, if we can make everything look like XML, then all the problems are solved". Or if we can make everything look like objects then everything is solved.

I believe that's a dead end because there will be more data models in the future, it doesn't scale. Imagine that tomorrow a completely new data model comes out, and you don't know if you can still map that to XML or Objects. The secret is to look at what mathematics has to offer you to solve this problem, and then try to translate this into something that normal programmers can understand. The way I explain it is like when you are designing a car. If you drive a car, you should know nothing about how the engine works, it should go smoothly; but if you are designing the car you have to know everything about physics of the engine and spark plugs and things like that. This is the same when you're designing this framework. We're using this theory from mathematics saying: "Can we abstract over any data model?" If once we can make it work for any data model, then you've solved the impedance mismatch, because then you are not dependant on any of these data models anymore.

L>>Ну это еще цветочки. Ты не видел какой мрак получается с бесконечными последовательностями.

using System;
using System.Collections.Generic;
using System.Linq;

class Program {
    static void Main(string[] args) {
        var arr = new[] { 1, 2, 3, 7, 8, 9, 10 };
        var q = arr
            .Zip(arr.Skip(1), (prev, curr) => new { curr, prev })
            .Select((item, i) => new { item.curr, item.prev, i })
            .Where(item => item.curr == item.prev + 1)
            .AttachState(
                () => new { include = true, prev = -1 },
                (item, prevState) => new {
                    include = !prevState.include || (prevState.prev + 1 != item.i),
                    prev = item.i
                },
                (item, state) => new { item.curr, state.include }
            )
            .Where(item => item.include)
            .Select(item => item.curr);

        foreach (var item in q) {
            Console.WriteLine(item);
        }
    }
}

public static class StateEnumerableExtensions {
    public static IEnumerable<TResult> AttachState<T, TState, TResult>(
        this IEnumerable<T> source,
        Func<TState> getInitialState,
        Func<T, TState, TState> getNextState,
        Func<T, TState, TResult> selector
    ) {
        var state = getInitialState();
        foreach (var item in source) {
            yield return selector(item, state);
            state = getNextState(item, state);
        }
    }
}

var arr  = new[] { 1, 2, 3, 7, 8, 9, 10 };
var list = new List<int>();
Func<int,bool> func = i => { list.Add(i); return true; };

var q1 =
    from a in arr.Select((n, i) => new { i, n })
    let prev = a.i - 1
    where
        prev >= 0 && a.n - 1 == arr[prev] && !list.Contains(prev) && func(a.i)
    select new { a.i, n = arr[a.i] };

foreach (var p in q1)
    Console.Write("{0} ", p.n);

var q2 =
    (from i in q1 select arr[i.i - 1]).
    Union(
    (from i in q1 where i.i + 1 < arr.Length select arr[i.i + 1])).
    OrderBy(i => i);

foreach (var p in q2)
    Console.Write("{0} ", p);

var q1 = arr
    .Where(((Func<Func<int, bool>>)(() => {
        bool first = true;
        int prev = 0;
        return i => {
            if (first) {
                first = false;
                prev = i;
                return false;
            } else if (i - prev == 1) {
                first = true;
                return true;
            } else {
                prev = 1;
                return false;
            }
        }; 
    }))());

public struct R
    {
        public TimeSpan time;
        public int value;
        public bool mark;
    }

var ro = Observable.FromEvent<REventArgs>(g, "R_arrived_event").Select(ea => ea.R);

List<R> buffer = new List<R> { new R() };
ro.Do(r =>
          {
              buffer.RemoveAll(rr => r.time - rr.time > TimeSpan.FromMilliseconds(1800));
              buffer.Add(r);
          })
    .Where(r => r.mark)
    .Subscribe(r => Console.WriteLine("mark = true at {0}, first within 1800 ms: {1}", r.time.TotalMilliseconds,
                                      buffer[0].time.TotalMilliseconds));

var rdata = new[]
                {
                    new R{mark = false,time = TimeSpan.FromMilliseconds(840), value = rnd.Next()},
                    new R{mark = false,time = TimeSpan.FromMilliseconds(928), value = rnd.Next()},
                    new R{mark = false,time = TimeSpan.FromMilliseconds(1100), value = rnd.Next()},
                    new R{mark = false,time = TimeSpan.FromMilliseconds(1657), value = rnd.Next()},
                    new R{mark = false,time = TimeSpan.FromMilliseconds(1799), value = rnd.Next()},
                    new R{mark = false,time = TimeSpan.FromMilliseconds(1948), value = rnd.Next()},
                    new R{mark = true,time = TimeSpan.FromMilliseconds(3332), value = rnd.Next()},
                    new R{mark = true, time = TimeSpan.FromMilliseconds(3900), value=rnd.Next()}
                };


var ro = Observable.Generate(0, // inital state 
                             index => index < rdata.Length, // continue 
                             index => rdata[index], // next value 
                             index =>
                                        {
                                            if (index == 0)
                                                return rdata[0].time;
                                            if (index == rdata.Length)
                                                return TimeSpan.FromMilliseconds(0);
                                            return rdata[index].time - rdata[index - 1].time;
                                        }, // waitInterval 
                             index => index + 1 // iterate 
);

func loop( c, o chan int ) // два раза типы писать не надо.
{ 
     prev := <- c          // := позволяет не писать мусора при объявлении+инициализации

     for val := range с    //читать из потока идиоматично в нашем случае так.
     { 
         if val != prev + 1       // это у нас типо фильтр входных значений, мы его четко отделили
         {     
            prev = val
            continue
         }
         
         o <- val                 //а здесь обработка результата
         prev = <- c              //и как положено, пропускаем один шаг.
                                  //Да, операция чтения - унарная, и надо писать "равно".
     }
}

A table is a multiset of rows. A row is a nonempty sequence of values. Every row of the same table has the same cardinality and contains a value of every column of that table.<...>The row is the smallest unit of data that can be inserted into a table and deleted from a table.

N>Да, использовать LINQ стали тут же не для РСУБД; но возникло оно именно как средство работы для РСУБД.

int pos = text.Split('\n').Take(line).Sum(s => s.Length + 1) + column;

LINQ по большому счету заточен под прикладную задачу, и нафиг не вперся в случае, если мы не работаем с реляционной БД.

static IEnumerable<int> Sample1C(IEnumerable<int> arr)
{
    bool first = true;
    int  prev  = 0;

    return arr.Where(i =>
    {
        if (first)
        {
            first = false;
            prev  = i;
        }
        else if (i - prev == 1)
        {
            first = true;
            return true;
        }
        else prev = i;

        return false;
    });
}

using System;
using System.Collections.Generic;
using System.Linq;

class Program {
    static void Main(string[] args) {
        var rs = GetData();
        var qrs = rs
            .AttachState(
                () => Enumerable.Empty<R>(),
                (r, prev) => prev.SkipWhile(_ => (r.Time - _.Time) > TimeSpan.FromMinutes(30)).Concat(new[] { r }),
                (r, state) => new { r, state }
            )
            .SkipWhile(_ => !_.r.Mark)
            .Select(_ => _.state)
            .First();

        foreach (var found in qrs)
            Console.WriteLine("{0}, {1}, {2}", found.Time, found.Value, found.Mark);
    }

    private static R[] GetData() {
        var rnd = new Random();
        var rs = new R[] { 
            new R {
                Time= DateTime.Today + new TimeSpan(0, 14, 00), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 15, 28), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 18, 20), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 27, 37), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 29, 59), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 32, 28), 
                Value = rnd.Next(100), 
                Mark = false 
            },
            new R {
                Time= DateTime.Today + new TimeSpan(0, 55, 32), 
                Value = rnd.Next(100), 
                Mark = true 
            },
        };
        return rs;
    }
}

public static class StateEnumerableExtensions {
    public static IEnumerable<TResult> AttachState<T, TState, TResult>(
        this IEnumerable<T> source,
        Func<TState> getInitialState,
        Func<T, TState, TState> getNextState,
        Func<T, TState, TResult> selector
    ) {
        var state = getInitialState();
        foreach (var item in source) {
            state = getNextState(item, state);
            yield return selector(item, state);
        }
    }
}

class R {
    public DateTime Time { get; set; }
    public int Value { get; set; }
    public bool Mark { get; set; }
}

            var array = new[] {1, 2, 3, 7, 8, 9, 10};

            var result = array.Aggregate(new 
                                         {
                                                PreviouslyTaken = array.FirstOrDefault() - 2,
                                                PreviousElement = array.FirstOrDefault() - 2,
                                                Result = (IEnumerable<int>)new int[0]
                                         },
                                         (ag, i) => new
                                                    {
                                                           PreviouslyTaken = (i - 1 == ag.PreviousElement) && ag.PreviousElement != ag.PreviouslyTaken 
                                                                             ? i : ag.PreviouslyTaken,
                                                           PreviousElement = i,
                                                           Result = (i - 1 == ag.PreviousElement && ag.PreviousElement != ag.PreviouslyTaken)
                                                                    ? ag.Result.Concat(new [] {i}) : ag.Result
                                                    },
                                         ag => ag.Result);

D_>            var array = new[] {1, 2, 3, 7, 8, 9, 10};

D_>            var result = array.Aggregate(new 
D_>                                         {
D_>                                                PreviouslyTaken = array.FirstOrDefault() - 2,
D_>                                                PreviousElement = array.FirstOrDefault() - 2,
D_>                                                Result = (IEnumerable<int>)new int[0]
D_>                                         },
D_>                                         (ag, i) => new
D_>                                                    {
D_>                                                           PreviouslyTaken = (i - 1 == ag.PreviousElement) && ag.PreviousElement != ag.PreviouslyTaken 
D_>                                                                             ? i : ag.PreviouslyTaken,
D_>                                                           PreviousElement = i,
D_>                                                           Result = (i - 1 == ag.PreviousElement && ag.PreviousElement != ag.PreviouslyTaken)
D_>                                                                    ? ag.Result.Concat(new [] {i}) : ag.Result
D_>                                                    },
D_>                                         ag => ag.Result);
D_>

var ary = new[] { 1, 2, 3, 7, 8, 9, 10 };

var result = ary.Aggregate(new { Prev = 0,   Skip = true,  Res = new List<int>() }, (acc, cur) => 
     acc.Skip            ? new { Prev = cur, Skip = false, acc.Res }
   : acc.Prev + 1 == cur ? new { Prev = cur, Skip = true,  Res = acc.Res.AddElem(cur) }
   :                       new { Prev = cur, Skip = false, acc.Res },
  ag => ag.Res);

foreach (var item in result)
  Console.WriteLine(item);

def lst = [1, 2, 3, 7, 8, 9, 10];

def (_, _, res) = lst.Aggregate((0, true, List()), ((prev, skip, res), cur) =>
  if      (skip)            (cur, false, res)
  else if (prev + 1 == cur) (cur,  true, { res.Add(cur); res })
  else                      (cur, false, res));

WriteLine($"..$res");

The original motivation behind LINQ was to address the conceptual and technical difficulties encountered when using databases with .NET programming languages. With LINQ, Microsoft’s intention was to provide a solution for the
problem of object-relational mapping, as well as to simplify the interaction
between objects and data sources. LINQ eventually evolved into a general-purpose
language-integrated querying toolset. [...]

func ( x []int ) p( i int ) bool {
    return i > 0 && x[ i - 1 ] == x[ i ] - 1
}

func gena_1( x []int ) []int {
    res := vector.NewIntVector(0)
    
    for i, val := range x {
        if x.p( i ) && !x.p( i - 1 ) {
            res.Push( val )
        }
    }

    return res.Data()
}

при этом извлечённый элемент не должен принимать участия в последующем сравнении

при этом предшествующий элемент не должен удовлетворять первому условию

Если идет речь о простоте выразительного средства, то есть языка — то разумеется тот, который без LINQ. Учитывая, что LINQ по большому счету заточен под прикладную задачу, и нафиг не вперся в случае, если мы не работаем с реляционной БД. И представляешь, в гугле при разработке веб-приложений как-то обходятся без РСУБД.

G>>К разговору твоя личность конечно не имеет никакого отношения, как и моя, но почему бы тебе не узнать об этом моем ощущении, правда? Не вижу причин, почему нет.

Rather than requiring the programmer to declare ahead of time that two types are related, in Go a type automatically satisfies any interface that specifies a subset of its methods.

VD>Сдается мне понятия простой и удобный весьма многогранны. Так что они ровным счетом ничего не выражают.

Для тебя — ничего. Для меня — выражают достаточно много. Спорить об этом не вижу смысла. Есть вещи, которые человек либо понимает, либо нет.

VD>Скажем я могу в C# 3.0 просто и удобно написать "запрос" к данным с помощью LINQ-а, а в более простом (в смысле примитивном) C# 2.0 не могу. Так какой же из этих вариантов проще?

G>Если идет речь о простоте выразительного средства, то есть языка — то разумеется тот, который без LINQ. Учитывая, что LINQ по большому счету заточен под прикладную задачу, и нафиг не вперся в случае, если мы не работаем с реляционной БД. И представляешь, в гугле при разработке веб-приложений как-то обходятся без РСУБД.

AVK>Пока что понты в топике больше от тебя